|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
public interface Flow<Config>
A Flow is a logical unit of work declared by an assembly of Pipe
instances connected to source
and sink Tap
instances.
FlowConnector
class
for supported platforms.
Note that Pipe
assemblies can be reused in multiple Flow instances. They maintain
no state regarding the Flow execution. Subsequently, Pipe
assemblies can be given
parameters through its calling Flow so they can be built in a generic fashion.
When a Flow is created, an optimized internal representation is created that is then executed
on the underlying execution platform. This is typically done by creating one or more FlowStep
instances.
Flows are submitted in order of dependency when used with a Cascade
. If two or more steps do not share the
same dependencies and all can be scheduled simultaneously, the getSubmitPriority()
value determines
the order in which all steps will be submitted for execution. The default submit priority is 5.
Use the FlowListener
to receive any events on the life-cycle of the Flow as it executes. Any
Tap
instances owned by the Flow also implementing FlowListener will automatically be added to the
set of listeners.
FlowListener
,
FlowConnector
Field Summary | |
---|---|
static String |
CASCADING_FLOW_ID
|
Method Summary | |
---|---|
void |
addListener(FlowListener flowListener)
Method addListener registers the given flowListener with this instance. |
void |
addStepListener(FlowStepListener flowStepListener)
Method addStepListener registers the given flowStepListener with this instance. |
boolean |
areSinksStale()
Method areSinksStale returns true if any of the sinks referenced are out of date in relation to the sources. |
boolean |
areSourcesNewer(long sinkModified)
Method areSourcesNewer returns true if any source is newer than the given sinkModified date value. |
void |
cleanup()
|
void |
complete()
Method complete starts the current Flow instance if it has not be previously started, then block until completion. |
String |
getCascadeID()
|
List<String> |
getCheckpointNames()
|
Map<String,Tap> |
getCheckpoints()
Method getCheckpoints returns the checkpoint taps of this Flow object. |
Collection<Tap> |
getCheckpointsCollection()
Method getCheckpointsCollection returns a Collection of checkpoint Tap s for this Flow object. |
Config |
getConfig()
Method getConfig returns the internal configuration object. |
Map<Object,Object> |
getConfigAsProperties()
Method getConfiAsProperties converts the internal configuration object into a Map of
key value pairs. |
Config |
getConfigCopy()
Method getConfigCopy returns a copy of the internal configuration object. |
Map<String,String> |
getFlowDescriptor()
Returns an immutable map of properties giving more details about the Flow object. |
FlowProcess<Config> |
getFlowProcess()
|
FlowSkipStrategy |
getFlowSkipStrategy()
Method getFlowSkipStrategy returns the current FlowSkipStrategy used by this Flow. |
FlowStats |
getFlowStats()
Method getFlowStats returns the flowStats of this Flow object. |
List<FlowStep<Config>> |
getFlowSteps()
Method getFlowSteps returns the flowSteps of this Flow object. |
FlowStepStrategy |
getFlowStepStrategy()
Returns the current FlowStepStrategy instance. |
String |
getID()
Method getID returns the ID of this Flow object. |
String |
getName()
Method getName returns the name of this Flow object. |
PlatformInfo |
getPlatformInfo()
|
String |
getProperty(String key)
|
String |
getRunID()
|
Tap |
getSink()
Method getSink returns the first sink of this Flow object. |
Tap |
getSink(String name)
|
long |
getSinkModified()
Method getSinkModified returns the youngest modified date of any sink Tap managed by this Flow instance. |
List<String> |
getSinkNames()
|
Map<String,Tap> |
getSinks()
Method getSinks returns the sinks of this Flow object. |
Collection<Tap> |
getSinksCollection()
Method getSinksCollection returns a Collection of sink Tap s for this Flow object. |
Tap |
getSource(String name)
|
List<String> |
getSourceNames()
|
Map<String,Tap> |
getSources()
Method getSources returns the sources of this Flow object. |
Collection<Tap> |
getSourcesCollection()
Method getSourcesCollection returns a Collection of source Tap s for this Flow object. |
int |
getSubmitPriority()
Method getSubmitPriority returns the submitPriority of this Flow object. |
String |
getTags()
|
List<String> |
getTrapNames()
|
Map<String,Tap> |
getTraps()
Method getTraps returns the traps of this Flow object. |
Collection<Tap> |
getTrapsCollection()
Method getTrapsCollection returns a Collection of trap Tap s for this Flow object. |
boolean |
hasListeners()
Method hasListeners returns true if FlowListener instances have been registered. |
boolean |
hasStepListeners()
Method hasStepListeners returns true if FlowStepListener instances have been registered
with any of the FlowStep s belonging to this instance |
boolean |
isSkipFlow()
Method isSkipFlow returns true if the parent Cascade should skip this Flow instance. |
boolean |
isStopJobsOnExit()
Method isStopJobsOnExit returns the stopJobsOnExit of this Flow object. |
TupleEntryIterator |
openSink()
Method openSink opens the first sink Tap. |
TupleEntryIterator |
openSink(String name)
Method openSink opens the named sink Tap. |
TupleEntryIterator |
openSource()
Method openSource opens the first source Tap. |
TupleEntryIterator |
openSource(String name)
Method openSource opens the named source Tap. |
TupleEntryIterator |
openTapForRead(Tap tap)
Method openTapForRead return a TupleEntryIterator for the given Tap instance. |
TupleEntryCollector |
openTapForWrite(Tap tap)
Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance. |
TupleEntryIterator |
openTrap()
Method openTrap opens the first trap Tap. |
TupleEntryIterator |
openTrap(String name)
Method openTrap opens the named trap Tap. |
void |
prepare()
Method prepare is used by a Cascade to notify the given Flow it should initialize or clear any resources
necessary for start() to be called successfully. |
boolean |
removeListener(FlowListener flowListener)
Method removeListener removes the given flowListener from this instance. |
boolean |
removeStepListener(FlowStepListener flowStepListener)
Method removeStepListener removes the given flowStepListener from this instance. |
boolean |
resourceExists(Tap tap)
Method resourceExists returns true if the resource represented by the given Tap instance exists. |
FlowSkipStrategy |
setFlowSkipStrategy(FlowSkipStrategy flowSkipStrategy)
Method setFlowSkipStrategy sets a new FlowSkipStrategy , the current strategy is returned. |
void |
setFlowStepStrategy(FlowStepStrategy flowStepStrategy)
Sets a default FlowStepStrategy instance. |
void |
setSubmitPriority(int submitPriority)
Method setSubmitPriority sets the submitPriority of this Flow object. |
void |
start()
Method start begins the execution of this Flow instance. |
boolean |
stepsAreLocal()
Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task. |
void |
stop()
Method stop stops all running jobs, killing any currently executing. |
void |
writeDOT(String filename)
Method writeDOT writes this Flow instance to the given filename as a DOT file for import into a graphics package. |
void |
writeStepsDOT(String filename)
Method writeStepsDOT writes this Flow step graph to the given filename as a DOT file for import into a graphics package. |
Methods inherited from interface cascading.management.UnitOfWork |
---|
getSpawnStrategy, getStats, setSpawnStrategy |
Field Detail |
---|
static final String CASCADING_FLOW_ID
Method Detail |
---|
String getName()
getName
in interface UnitOfWork<FlowStats>
void prepare()
Cascade
to notify the given Flow it should initialize or clear any resources
necessary for start()
to be called successfully.
Specifically, this implementation calls BaseFlow.deleteSinksIfNotUpdate()
&& BaseFlow.deleteTrapsIfNotUpdate()
.
prepare
in interface UnitOfWork<FlowStats>
IOException
- whenvoid start()
complete()
to block until this Flow completes.
start
in interface UnitOfWork<FlowStats>
void stop()
stop
in interface UnitOfWork<FlowStats>
void complete()
complete
in interface UnitOfWork<FlowStats>
void cleanup()
cleanup
in interface UnitOfWork<FlowStats>
Config getConfig()
FlowConnector
for setting
default properties visible to children. Or see FlowStepStrategy
for setting properties on
individual steps before they are executed.
Config getConfigCopy()
Map<Object,Object> getConfigAsProperties()
Map
of
key value pairs.
String getProperty(String key)
String getID()
getID
in interface UnitOfWork<FlowStats>
Map<String,String> getFlowDescriptor()
FlowDef.addDescription(String, String)
to set values on a given Flow.
Flow descriptions provide meta-data to monitoring systems describing the workload a given Flow represents.
For known description types, see FlowDescriptors
.
String getTags()
getTags
in interface UnitOfWork<FlowStats>
int getSubmitPriority()
void setSubmitPriority(int submitPriority)
submitPriority
- the submitPriority of this FlowStep object.FlowProcess<Config> getFlowProcess()
FlowStats getFlowStats()
boolean hasListeners()
FlowListener
instances have been registered.
void addListener(FlowListener flowListener)
flowListener
- of type FlowListenerboolean removeListener(FlowListener flowListener)
flowListener
- of type FlowListener
boolean hasStepListeners()
FlowStepListener
instances have been registered
with any of the FlowStep
s belonging to this instance
void addStepListener(FlowStepListener flowStepListener)
flowStepListener
- of type addStepListenerboolean removeStepListener(FlowStepListener flowStepListener)
flowStepListener
- of type FlowStepListener
FlowStep
belonging to this instanceMap<String,Tap> getSources()
List<String> getSourceNames()
Tap getSource(String name)
Collection<Tap> getSourcesCollection()
Collection
of source Tap
s for this Flow object.
Map<String,Tap> getSinks()
List<String> getSinkNames()
Tap getSink(String name)
Collection<Tap> getSinksCollection()
Collection
of sink Tap
s for this Flow object.
Tap getSink()
Map<String,Tap> getTraps()
List<String> getTrapNames()
Collection<Tap> getTrapsCollection()
Collection
of trap Tap
s for this Flow object.
Map<String,Tap> getCheckpoints()
List<String> getCheckpointNames()
Collection<Tap> getCheckpointsCollection()
Collection
of checkpoint Tap
s for this Flow object.
FlowSkipStrategy getFlowSkipStrategy()
FlowSkipStrategy
used by this Flow.
FlowSkipStrategy setFlowSkipStrategy(FlowSkipStrategy flowSkipStrategy)
FlowSkipStrategy
, the current strategy is returned.
FlowSkipStrategy instances define when a Flow instance should be skipped. The default strategy is FlowSkipIfSinkNotStale
.
An alternative strategy would be FlowSkipIfSinkExists
.
A FlowSkipStrategy will not be consulted when executing a Flow directly through start()
or complete()
. Only
when the Flow is executed through a Cascade
instance.
flowSkipStrategy
- of type FlowSkipStrategy
boolean isSkipFlow() throws IOException
Cascade
should skip this Flow instance. True is returned
if the current FlowSkipStrategy
returns true.
IOException
- whenboolean areSinksStale() throws IOException
Tap.isReplace()
returns true.
IOException
- whenboolean areSourcesNewer(long sinkModified) throws IOException
sinkModified
- of type long
IOException
- whenlong getSinkModified() throws IOException
Tap
managed by this Flow instance.
If zero (0) is returned, at least one of the sink resources does not exist. If minus one (-1) is returned,
atleast one of the sinks are marked for delete (returns true
).
IOException
- whenFlowStepStrategy getFlowStepStrategy()
FlowStepStrategy
instance.
void setFlowStepStrategy(FlowStepStrategy flowStepStrategy)
FlowStepStrategy
instance.
Use a FlowStepStrategy to change FlowStep
configuration properties
before the properties are submitted to the underlying platform for the step
unit of work.
flowStepStrategy
- The FlowStepStrategy to use.List<FlowStep<Config>> getFlowSteps()
TupleEntryIterator openSource() throws IOException
IOException
- whenTupleEntryIterator openSource(String name) throws IOException
name
- of type String
IOException
- whenTupleEntryIterator openSink() throws IOException
IOException
- whenTupleEntryIterator openSink(String name) throws IOException
name
- of type String
IOException
- whenTupleEntryIterator openTrap() throws IOException
IOException
- whenTupleEntryIterator openTrap(String name) throws IOException
name
- of type String
IOException
- whenboolean resourceExists(Tap tap) throws IOException
tap
- of type Tap
IOException
- whenTupleEntryIterator openTapForRead(Tap tap) throws IOException
TupleEntryIterator
for the given Tap instance.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.
tap
- of type Tap
IOException
- when there is an error opening the resourceTupleEntryCollector openTapForWrite(Tap tap) throws IOException
tap
- of type Tap
IOException
- when there is an error opening the resourcevoid writeDOT(String filename)
filename
- of type Stringvoid writeStepsDOT(String filename)
filename
- of type StringString getCascadeID()
String getRunID()
PlatformInfo getPlatformInfo()
boolean stepsAreLocal()
boolean isStopJobsOnExit()
true
.
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |