public class ProcessFlow<Process,Config> extends BaseFlow<Config>
Flow
subclass that supports custom Riffle jobs.
Use this class to allow custom Riffle jobs to participate in the Cascade
scheduler. If
other Flow instances in the Cascade share resources with this Flow instance, all participants will be scheduled
according to their dependencies (topologically).
Currently FlowListener
s are supported but the
FlowListener.onThrowable(cascading.flow.Flow, Throwable)
event is not.BaseFlow.FlowHolder
flowStats, shutdownHook, sinks, sources, stop, stopJobsOnExit, thread
CASCADING_FLOW_ID
NULL
Constructor and Description |
---|
ProcessFlow(Map<Object,Object> properties,
String name,
Process process)
Constructor ProcessFlow creates a new ProcessFlow instance.
|
ProcessFlow(Map<Object,Object> properties,
String name,
Process process,
Map<String,String> flowDescriptor)
Constructor ProcessFlow creates a new ProcessFlow instance.
|
ProcessFlow(String name,
Process process)
Constructor ProcessFlow creates a new ProcessFlow instance.
|
Modifier and Type | Method and Description |
---|---|
void |
cleanup() |
void |
complete()
Method complete starts the current Flow instance if it has not be previously started, then block until completion.
|
Config |
getConfig()
Method getConfig returns the internal configuration object.
|
Map<Object,Object> |
getConfigAsProperties()
Method getConfigAsProperties converts the internal configuration object into a
Map of
key value pairs. |
Config |
getConfigCopy()
Method getConfigCopy returns a copy of the internal configuration object.
|
FlowProcess<Config> |
getFlowProcess() |
protected int |
getMaxNumParallelSteps() |
Process |
getProcess()
Method getProcess returns the process of this ProcessFlow object.
|
String |
getProperty(String key)
Returns the String property associated with the given key from the current Configuration instance.
|
protected void |
initConfig(Map<Object,Object> properties,
Config parentConfig)
This method creates a new internal Config with the parentConfig as defaults using the properties to override
the defaults.
|
protected void |
internalClean(boolean stop) |
protected void |
internalShutdown() |
protected void |
internalStart() |
protected Config |
newConfig(Config defaultConfig) |
void |
prepare()
Method prepare is used by a
Cascade to notify the given Flow it should initialize or clear any resources
necessary for Flow.start() to be called successfully. |
protected void |
setConfigProperty(Config properties,
Object key,
Object value) |
void |
setTapFromProcess()
Method setTapFromProcess build
Tap instance for the give process incoming and outgoing dependencies. |
void |
start()
Method start begins the execution of this Flow instance.
|
boolean |
stepsAreLocal()
Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.
|
void |
stop()
Method stop stops all running jobs, killing any currently executing.
|
String |
toString() |
addListener, addPlannerProperties, addStepListener, areSinksStale, areSourcesNewer, createConfig, createFlowThread, deleteCheckpointsIfNotUpdate, deleteCheckpointsIfReplace, deleteSinks, deleteSinksIfNotUpdate, deleteSinksIfReplace, deleteTrapsIfNotUpdate, deleteTrapsIfReplace, fireOnCompleted, fireOnStarting, fireOnStopping, fireOnThrowable, fireOnThrowable, getCascadeID, getCascadingServices, getCheckpointNames, getCheckpoints, getCheckpointsCollection, getClassPath, getFieldsFor, getFlowDescriptor, getFlowSession, getFlowSkipStrategy, getFlowStats, getFlowSteps, getFlowStepStrategy, getHolder, getID, getName, getPlannerInfo, getPlatformInfo, getRunID, getSink, getSink, getSinkModified, getSinkNames, getSinks, getSinksCollection, getSource, getSourceNames, getSources, getSourcesCollection, getSpawnStrategy, getStats, getSubmitPriority, getTags, getTotalSliceCPUMilliSeconds, getTrapNames, getTraps, getTrapsCollection, handleExecutorShutdown, hasListeners, hasStepListeners, initFromProperties, initialize, initializeNewJobsMap, initSteps, internalStopAllJobs, isDebugEnabled, isInfoEnabled, isSkipFlow, isStopJobsOnExit, logDebug, logError, logError, logInfo, logWarn, logWarn, logWarn, openSink, openSink, openSource, openSource, openTapForRead, openTapForWrite, openTrap, openTrap, presentSinkFields, presentSourceFields, registerShutdownHook, removeListener, removeStepListener, resourceExists, retrieveSinkFields, retrieveSourceFields, setCascade, setCheckpoints, setFlowSkipStrategy, setFlowStepGraph, setFlowStepStrategy, setName, setPlannerInfo, setSinks, setSources, setSpawnStrategy, setSubmitPriority, setTraps, updateSchemes, writeDOT, writeStepsDOT
@ConstructorProperties(value={"name","process"}) public ProcessFlow(String name, Process process)
name
- of type Stringprocess
- of type JobConf@ConstructorProperties(value={"properties","name","process"}) public ProcessFlow(Map<Object,Object> properties, String name, Process process)
properties
- of type Mapname
- of type Stringprocess
- of type P@ConstructorProperties(value={"properties","name","process","flowDescriptor"}) public ProcessFlow(Map<Object,Object> properties, String name, Process process, Map<String,String> flowDescriptor)
properties
- of type Mapname
- of type Stringprocess
- of type PflowDescriptor
- pf type LinkedHashMappublic void setTapFromProcess()
Tap
instance for the give process incoming and outgoing dependencies.
This method may be called repeatedly to re-configure the source and sink taps.public Process getProcess()
protected void initConfig(Map<Object,Object> properties, Config parentConfig)
BaseFlow
initConfig
in class BaseFlow<Config>
properties
- of type MapparentConfig
- of type Configprotected void setConfigProperty(Config properties, Object key, Object value)
setConfigProperty
in class BaseFlow<Config>
public Config getConfig()
Flow
FlowConnector
for setting
default properties visible to children. Or see FlowStepStrategy
for setting properties on
individual steps before they are executed.public Config getConfigCopy()
Flow
public Map<Object,Object> getConfigAsProperties()
Flow
Map
of
key value pairs.public String getProperty(String key)
Flow
key
- of type Stringpublic FlowProcess<Config> getFlowProcess()
public boolean stepsAreLocal()
Flow
public void prepare()
Flow
Cascade
to notify the given Flow it should initialize or clear any resources
necessary for Flow.start()
to be called successfully.
Specifically, this implementation calls BaseFlow.deleteSinksIfNotUpdate()
&& BaseFlow.deleteTrapsIfNotUpdate()
.public void start()
Flow
Flow.complete()
to block until this Flow completes.protected void internalStart()
internalStart
in class BaseFlow<Config>
public void stop()
Flow
protected void internalClean(boolean stop)
internalClean
in class BaseFlow<Config>
public void complete()
Flow
public void cleanup()
protected int getMaxNumParallelSteps()
getMaxNumParallelSteps
in class BaseFlow<Config>
protected void internalShutdown()
internalShutdown
in class BaseFlow<Config>
Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.