cascading.flow.process

Class ProcessFlow<Process,Config>

java.lang.Object

cascading.flow.BaseFlow<Config>

cascading.flow.process.ProcessFlow<Process,Config>

All Implemented Interfaces:

Flow<Config>, UnitOfWork<FlowStats>, ProcessLogger

public class ProcessFlow<Process,Config>
extends BaseFlow<Config>

Class ProcessFlow is a Flow subclass that supports custom Riffle jobs.

Use this class to allow custom Riffle jobs to participate in the Cascade scheduler. If other Flow instances in the Cascade share resources with this Flow instance, all participants will be scheduled according to their dependencies (topologically).

Currently FlowListeners are supported but the FlowListener.onThrowable(cascading.flow.Flow, Throwable) event is not.

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.BaseFlow

BaseFlow.FlowHolder

Field Summary

Fields inherited from class cascading.flow.BaseFlow

completed, flowCanonicalHash, flowDescriptor, flowElementGraph, flowStats, flowStepGraph, platformInfo, shutdownHook, sinks, sources, steps, stop, stopJobsOnExit, thread, throwable

Fields inherited from interface cascading.flow.Flow

CASCADING_FLOW_ID

Fields inherited from interface cascading.util.ProcessLogger

NULL

Constructor Summary

Constructors

Constructor and Description
ProcessFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, Process process) Constructor ProcessFlow creates a new ProcessFlow instance.
ProcessFlow(java.util.Map<java.lang.Object,java.lang.Object> properties, java.lang.String name, Process process, java.util.Map<java.lang.String,java.lang.String> flowDescriptor) Constructor ProcessFlow creates a new ProcessFlow instance.
ProcessFlow(java.lang.String name, Process process) Constructor ProcessFlow creates a new ProcessFlow instance.

Method Summary

Modifier and Type	Method and Description
void	cleanup()
void	complete() Method complete starts the current Flow instance if it has not be previously started, then block until completion.
Config	getConfig() Method getConfig returns the internal configuration object.
java.util.Map<java.lang.Object,java.lang.Object>	getConfigAsProperties() Method getConfigAsProperties converts the internal configuration object into a Map of key value pairs.
Config	getConfigCopy() Method getConfigCopy returns a copy of the internal configuration object.
FlowProcess<Config>	getFlowProcess()
protected int	getMaxNumParallelSteps()
Process	getProcess() Method getProcess returns the process of this ProcessFlow object.
java.lang.String	getProperty(java.lang.String key) Returns the String property associated with the given key from the current Configuration instance.
protected void	initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties, Config parentConfig) This method creates a new internal Config with the parentConfig as defaults using the properties to override the defaults.
protected void	internalClean(boolean stop)
protected void	internalShutdown()
protected void	internalStart()
protected Config	newConfig(Config defaultConfig)
void	prepare() Method prepare is used by a Cascade to notify the given Flow it should initialize or clear any resources necessary for Flow.start() to be called successfully.
protected void	setConfigProperty(Config properties, java.lang.Object key, java.lang.Object value)
void	setTapFromProcess() Method setTapFromProcess build Tap instance for the give process incoming and outgoing dependencies.
void	start() Method start begins the execution of this Flow instance.
boolean	stepsAreLocal() Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.
void	stop() Method stop stops all running jobs, killing any currently executing.
java.lang.String	toString()

Methods inherited from class cascading.flow.BaseFlow

addListener, addPlannerProperties, addSessionProperties, addStepListener, areSinksStale, areSourcesNewer, createConfig, createFlowCanonicalHash, createFlowStats, createFlowThread, createPrepareFlowStats, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getClientState, getEligibleJobsSize, getFieldsFor, getFlowCanonicalHash, getFlowDescriptor, getFlowElementGraph, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowStepGraph, getFlowSteps, getFlowStepStrategy, getHolder, getID, getJobMapCallables, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTotalSliceCPUMilliSeconds, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, ifStoppingBlockUntilComplete, initFromProperties, initialize, initializeChildStats, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isJobsMapInitialized, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeListeners, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowElementGraph, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, spawnSteps, updateJobsMap, updateSchemes, writeDOT, writeStepsDOT

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface cascading.flow.Flow

getSinkEntryStream, getSinkEntryStream, getSinkEntryStream, getSinkEntryStream, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkEntryStreamCopy, getSinkTupleStream, getSinkTupleStream, getSinkTupleStream, getSinkTupleStream, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSinkTupleStreamCopy, getSourceEntryStream, getSourceEntryStream, getSourceEntryStreamCopy, getSourceEntryStreamCopy, getSourceTupleStream, getSourceTupleStream, getSourceTupleStreamCopy, getSourceTupleStreamCopy

Constructor Detail

ProcessFlow

@ConstructorProperties(value={"name","process"})
public ProcessFlow(java.lang.String name,
                                                                    Process process)

Constructor ProcessFlow creates a new ProcessFlow instance.

Parameters:

name - of type String

process - of type JobConf

ProcessFlow

@ConstructorProperties(value={"properties","name","process"})
public ProcessFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                                                                 java.lang.String name,
                                                                                 Process process)

Constructor ProcessFlow creates a new ProcessFlow instance.

Parameters:

properties - of type Map

name - of type String

process - of type P

ProcessFlow

@ConstructorProperties(value={"properties","name","process","flowDescriptor"})
public ProcessFlow(java.util.Map<java.lang.Object,java.lang.Object> properties,
                                                                                                  java.lang.String name,
                                                                                                  Process process,
                                                                                                  java.util.Map<java.lang.String,java.lang.String> flowDescriptor)

Constructor ProcessFlow creates a new ProcessFlow instance.

Parameters:

properties - of type Map

name - of type String

process - of type P

flowDescriptor - pf type LinkedHashMap

Method Detail

setTapFromProcess

public void setTapFromProcess()

Method setTapFromProcess build Tap instance for the give process incoming and outgoing dependencies.

This method may be called repeatedly to re-configure the source and sink taps.

getProcess

public Process getProcess()

Method getProcess returns the process of this ProcessFlow object.

Returns:

the process (type P) of this ProcessFlow object.

initConfig

protected void initConfig(java.util.Map<java.lang.Object,java.lang.Object> properties,
                          Config parentConfig)

Description copied from class: BaseFlow

This method creates a new internal Config with the parentConfig as defaults using the properties to override the defaults.

Specified by:

initConfig in class BaseFlow<Config>

Parameters:

properties - of type Map

parentConfig - of type Config

setConfigProperty

protected void setConfigProperty(Config properties,
                                 java.lang.Object key,
                                 java.lang.Object value)

Specified by:

setConfigProperty in class BaseFlow<Config>

newConfig

protected Config newConfig(Config defaultConfig)

Specified by:

newConfig in class BaseFlow<Config>

getConfig

public Config getConfig()

Description copied from interface: Flow

Method getConfig returns the internal configuration object.

Any changes to this object will not be reflected in child steps. See FlowConnector for setting default properties visible to children. Or see FlowStepStrategy for setting properties on individual steps before they are executed.

Returns:

the default configuration of this Flow

getConfigCopy

public Config getConfigCopy()

Description copied from interface: Flow

Method getConfigCopy returns a copy of the internal configuration object. This object can be safely modified.

Returns:

a copy of the default configuration of this Flow

getConfigAsProperties

public java.util.Map<java.lang.Object,java.lang.Object> getConfigAsProperties()

Description copied from interface: Flow

Method getConfigAsProperties converts the internal configuration object into a Map of key value pairs.

Returns:

a Map of key/value pairs

getProperty

public java.lang.String getProperty(java.lang.String key)

Description copied from interface: Flow

Returns the String property associated with the given key from the current Configuration instance.

Parameters:

key - of type String

Returns:

the String value

getFlowProcess

public FlowProcess<Config> getFlowProcess()

stepsAreLocal

public boolean stepsAreLocal()

Description copied from interface: Flow

Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.

Returns:

boolean

prepare

public void prepare()

Description copied from interface: Flow

Method prepare is used by a Cascade to notify the given Flow it should initialize or clear any resources necessary for Flow.start() to be called successfully.

Specifically, this implementation calls BaseFlow.deleteSinksIfNotUpdate() && BaseFlow.deleteTrapsIfNotUpdate().

Specified by:

prepare in interface Flow<Config>

Specified by:

prepare in interface UnitOfWork<FlowStats>

Overrides:

prepare in class BaseFlow<Config>

start

public void start()

Description copied from interface: Flow

Method start begins the execution of this Flow instance. It will return immediately. Use the method Flow.complete() to block until this Flow completes.

Specified by:

start in interface Flow<Config>

Specified by:

start in interface UnitOfWork<FlowStats>

Overrides:

start in class BaseFlow<Config>

internalStart

protected void internalStart()

Specified by:

internalStart in class BaseFlow<Config>

stop

public void stop()

Description copied from interface: Flow

Method stop stops all running jobs, killing any currently executing.

Specified by:

stop in interface Flow<Config>

Specified by:

stop in interface UnitOfWork<FlowStats>

Overrides:

stop in class BaseFlow<Config>

internalClean

protected void internalClean(boolean stop)

Specified by:

internalClean in class BaseFlow<Config>

complete

public void complete()

Description copied from interface: Flow

Method complete starts the current Flow instance if it has not be previously started, then block until completion.

Specified by:

complete in interface Flow<Config>

Specified by:

complete in interface UnitOfWork<FlowStats>

Overrides:

complete in class BaseFlow<Config>

cleanup

public void cleanup()

Specified by:

cleanup in interface Flow<Config>

Specified by:

cleanup in interface UnitOfWork<FlowStats>

Overrides:

cleanup in class BaseFlow<Config>

getMaxNumParallelSteps

protected int getMaxNumParallelSteps()

Specified by:

getMaxNumParallelSteps in class BaseFlow<Config>

internalShutdown

protected void internalShutdown()

Specified by:

internalShutdown in class BaseFlow<Config>

toString

public java.lang.String toString()

Overrides:

toString in class BaseFlow<Config>