cascading.flow.tez

Class Hadoop2TezFlow

java.lang.Object

cascading.flow.BaseFlow<TezConfiguration>

cascading.flow.tez.Hadoop2TezFlow

All Implemented Interfaces:

Flow<TezConfiguration>, UnitOfWork<FlowStats>, ProcessLogger

public class Hadoop2TezFlow
extends BaseFlow<TezConfiguration>

Class HadoopFlow is the Apache Hadoop specific implementation of a Flow.

HadoopFlow must be created through a Hadoop2TezFlowConnector instance.

If classpath paths are provided on the FlowDef, the Hadoop distributed cache mechanism will be used to augment the remote classpath.

Any path elements that are relative will be uploaded to HDFS, and the HDFS URI will be used on the JobConf. Note all paths are added as "files" to the JobConf, not archives, so they aren't needlessly uncompressed cluster side.

See Also:

Hadoop2TezFlowConnector

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.BaseFlow

BaseFlow.FlowHolder

Field Summary

Fields inherited from class cascading.flow.BaseFlow

completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable

Fields inherited from interface cascading.flow.Flow

CASCADING_FLOW_ID

Fields inherited from interface cascading.util.ProcessLogger

NULL

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Hadoop2TezFlow()
	Hadoop2TezFlow(PlatformInfo platformInfo, java.util.Map<java.lang.Object,java.lang.Object> properties, TezConfiguration flowConf, FlowDef flowDef)

Method Summary

Modifier and Type	Method and Description
TezConfiguration	getConfig()
java.util.Map<java.lang.Object,java.lang.Object>	getConfigAsProperties()
TezConfiguration	getConfigCopy()
FlowProcess<TezConfiguration>	getFlowProcess()
java.lang.String	getFlowStagingPath()
protected int	getMaxNumParallelSteps()
java.lang.String	getProperty(java.lang.String key) Method getProperty returns the value associated with the given key from the underlying properties system.
protected long	getTotalSliceCPUMilliSeconds()
protected void	initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties, TezConfiguration parentConfig)
protected void	initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)
protected void	internalClean(boolean stop)
protected void	internalShutdown()
protected void	internalStart()
boolean	isPreserveTemporaryFiles() Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.
protected TezConfiguration	newConfig(TezConfiguration defaultConfig)
protected void	setConfigProperty(TezConfiguration config, java.lang.Object key, java.lang.Object value)
boolean	stepsAreLocal()

Methods inherited from class cascading.flow.BaseFlow

addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, cleanup, complete, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, ifStoppingBlockUntilComplete, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, prepare, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, start, stop, toString, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface cascading.flow.Flow

getSinkEntryStream, getSinkEntryStream, getSinkEntryStream, getSinkEntryStream, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkTupleStream, getSinkTupleStream, getSinkTupleStream, getSinkTupleStream, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSourceEntryStream, getSourceEntryStream, getSourceEntryStreamCopy, getSourceEntryStreamCopy, getSourceTupleStream, getSourceTupleStream, getSourceTupleStreamCopy, getSourceTupleStreamCopy

Constructor Detail

Hadoop2TezFlow

protected Hadoop2TezFlow()

Hadoop2TezFlow

public Hadoop2TezFlow(PlatformInfo platformInfo,
                      java.util.Map<java.lang.Object,java.lang.Object> properties,
                      TezConfiguration flowConf,
                      FlowDef flowDef)

Method Detail

initFromProperties

protected void initFromProperties(java.util.Map<java.lang.Object,java.lang.Object> properties)

Overrides:

initFromProperties in class BaseFlow<TezConfiguration>

initConfig

protected void initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties,
                          TezConfiguration parentConfig)

Specified by:

initConfig in class BaseFlow<TezConfiguration>

getFlowStagingPath

public java.lang.String getFlowStagingPath()

setConfigProperty

protected void setConfigProperty(TezConfiguration config,
                                 java.lang.Object key,
                                 java.lang.Object value)

Specified by:

setConfigProperty in class BaseFlow<TezConfiguration>

newConfig

protected TezConfiguration newConfig(TezConfiguration defaultConfig)

Specified by:

newConfig in class BaseFlow<TezConfiguration>

getConfig

@ProcessConfiguration
public TezConfiguration getConfig()

getConfigCopy

public TezConfiguration getConfigCopy()

getConfigAsProperties

public java.util.Map<java.lang.Object,java.lang.Object> getConfigAsProperties()

getProperty

public java.lang.String getProperty(java.lang.String key)

Method getProperty returns the value associated with the given key from the underlying properties system.

Parameters:

key - of type String

Returns:

String

getFlowProcess

public FlowProcess<TezConfiguration> getFlowProcess()

isPreserveTemporaryFiles

public boolean isPreserveTemporaryFiles()

Method isPreserveTemporaryFiles returns false if temporary files will be cleaned when this Flow completes.

Returns:

the preserveTemporaryFiles (type boolean) of this Flow object.

internalStart

protected void internalStart()

Specified by:

internalStart in class BaseFlow<TezConfiguration>

stepsAreLocal

public boolean stepsAreLocal()

internalClean

protected void internalClean(boolean stop)

Specified by:

internalClean in class BaseFlow<TezConfiguration>

internalShutdown

protected void internalShutdown()

Specified by:

internalShutdown in class BaseFlow<TezConfiguration>

getMaxNumParallelSteps

protected int getMaxNumParallelSteps()

Specified by:

getMaxNumParallelSteps in class BaseFlow<TezConfiguration>

getTotalSliceCPUMilliSeconds

protected long getTotalSliceCPUMilliSeconds()

Overrides:

getTotalSliceCPUMilliSeconds in class BaseFlow<TezConfiguration>