public interface Flow<Config> extends UnitOfWork<FlowStats>
Pipe
instances connected to source
and sink Tap
instances.
A Flow is then executed to push the incoming source data through the assembly into one or more sinks.
A Flow sub-class instance may not be instantiated directly in most cases, see sub-classes of FlowConnector
class
for supported platforms.
Note that Pipe
assemblies can be reused in multiple Flow instances. They maintain
no state regarding the Flow execution. Subsequently, Pipe
assemblies can be given
parameters through its calling Flow so they can be built in a generic fashion.
When a Flow is created, an optimized internal representation is created that is then executed
on the underlying execution platform. This is typically done by creating one or more FlowStep
instances.
Flows are submitted in order of dependency when used with a Cascade
. If two or more steps do not share the
same dependencies and all can be scheduled simultaneously, the getSubmitPriority()
value determines
the order in which all steps will be submitted for execution. The default submit priority is 5.
Use the FlowListener
to receive any events on the life-cycle of the Flow as it executes. Any
Tap
instances owned by the Flow also implementing FlowListener will automatically be added to the
set of listeners.FlowListener
,
FlowConnector
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
CASCADING_FLOW_ID |
Modifier and Type | Method and Description |
---|---|
void |
addListener(FlowListener flowListener)
Method addListener registers the given flowListener with this instance.
|
void |
addStepListener(FlowStepListener flowStepListener)
Method addStepListener registers the given flowStepListener with this instance.
|
boolean |
areSinksStale()
Method areSinksStale returns true if any of the sinks referenced are out of date in relation to the sources.
|
boolean |
areSourcesNewer(long sinkModified)
Method areSourcesNewer returns true if any source is newer than the given sinkModified date value.
|
void |
cleanup() |
void |
complete()
Method complete starts the current Flow instance if it has not be previously started, then block until completion.
|
java.lang.String |
getCascadeID()
Returns the parent Cascade ID that owns this Flow instance.
|
java.util.List<java.lang.String> |
getCheckpointNames() |
java.util.Map<java.lang.String,Tap> |
getCheckpoints()
Method getCheckpoints returns the checkpoint taps of this Flow object.
|
java.util.Collection<Tap> |
getCheckpointsCollection()
Method getCheckpointsCollection returns a
Collection of checkpoint Tap s for this Flow object. |
Config |
getConfig()
Method getConfig returns the internal configuration object.
|
java.util.Map<java.lang.Object,java.lang.Object> |
getConfigAsProperties()
Method getConfigAsProperties converts the internal configuration object into a
Map of
key value pairs. |
Config |
getConfigCopy()
Method getConfigCopy returns a copy of the internal configuration object.
|
java.util.Map<java.lang.String,java.lang.String> |
getFlowDescriptor()
Returns an immutable map of properties giving more details about the Flow object.
|
FlowProcess<Config> |
getFlowProcess() |
FlowSkipStrategy |
getFlowSkipStrategy()
Method getFlowSkipStrategy returns the current
FlowSkipStrategy used by this Flow. |
FlowStats |
getFlowStats()
Method getFlowStats returns the flowStats of this Flow object.
|
java.util.List<FlowStep<Config>> |
getFlowSteps()
Method getFlowSteps returns the flowSteps of this Flow object.
|
FlowStepStrategy |
getFlowStepStrategy()
Returns the current
FlowStepStrategy instance. |
java.lang.String |
getID()
Method getID returns the ID of this Flow object.
|
java.lang.String |
getName()
Method getName returns the name of this Flow object.
|
PlannerInfo |
getPlannerInfo()
Returns any meta-data about the planner that created this Flow instance.
|
PlatformInfo |
getPlatformInfo()
Returns any meta-data about the underlying platform this Flow instance will run against.
|
java.lang.String |
getProperty(java.lang.String key)
Returns the String property associated with the given key from the current Configuration instance.
|
java.lang.String |
getRunID()
Returns the run ID given when this Flow instance was defined in the FlowDef.
|
Tap |
getSink()
Method getSink returns the first sink of this Flow object.
|
Tap |
getSink(java.lang.String name) |
long |
getSinkModified()
Method getSinkModified returns the youngest modified date of any sink
Tap managed by this Flow instance. |
java.util.List<java.lang.String> |
getSinkNames() |
java.util.Map<java.lang.String,Tap> |
getSinks()
Method getSinks returns the sinks of this Flow object.
|
java.util.Collection<Tap> |
getSinksCollection()
Method getSinksCollection returns a
Collection of sink Tap s for this Flow object. |
Tap |
getSource(java.lang.String name) |
java.util.List<java.lang.String> |
getSourceNames() |
java.util.Map<java.lang.String,Tap> |
getSources()
Method getSources returns the sources of this Flow object.
|
java.util.Collection<Tap> |
getSourcesCollection()
Method getSourcesCollection returns a
Collection of source Tap s for this Flow object. |
int |
getSubmitPriority()
Method getSubmitPriority returns the submitPriority of this Flow object.
|
java.lang.String |
getTags() |
java.util.List<java.lang.String> |
getTrapNames() |
java.util.Map<java.lang.String,Tap> |
getTraps()
Method getTraps returns the traps of this Flow object.
|
java.util.Collection<Tap> |
getTrapsCollection()
Method getTrapsCollection returns a
Collection of trap Tap s for this Flow object. |
boolean |
hasListeners()
Method hasListeners returns true if
FlowListener instances have been registered. |
boolean |
hasStepListeners()
Method hasStepListeners returns true if
FlowStepListener instances have been registered
with any of the FlowStep s belonging to this instance |
boolean |
isSkipFlow()
Method isSkipFlow returns true if the parent
Cascade should skip this Flow instance. |
boolean |
isStopJobsOnExit()
Method isStopJobsOnExit returns the stopJobsOnExit of this Flow object.
|
TupleEntryIterator |
openSink()
Method openSink opens the first sink Tap.
|
TupleEntryIterator |
openSink(java.lang.String name)
Method openSink opens the named sink Tap.
|
TupleEntryIterator |
openSource()
Method openSource opens the first source Tap.
|
TupleEntryIterator |
openSource(java.lang.String name)
Method openSource opens the named source Tap.
|
TupleEntryIterator |
openTapForRead(Tap tap)
Method openTapForRead return a
TupleEntryIterator for the given Tap instance. |
TupleEntryCollector |
openTapForWrite(Tap tap)
Method openTapForWrite returns a (@link TupleCollector} for the given Tap instance.
|
TupleEntryIterator |
openTrap()
Method openTrap opens the first trap Tap.
|
TupleEntryIterator |
openTrap(java.lang.String name)
Method openTrap opens the named trap Tap.
|
void |
prepare()
|
boolean |
removeListener(FlowListener flowListener)
Method removeListener removes the given flowListener from this instance.
|
boolean |
removeStepListener(FlowStepListener flowStepListener)
Method removeStepListener removes the given flowStepListener from this instance.
|
boolean |
resourceExists(Tap tap)
Method resourceExists returns true if the resource represented by the given Tap instance exists.
|
FlowSkipStrategy |
setFlowSkipStrategy(FlowSkipStrategy flowSkipStrategy)
Method setFlowSkipStrategy sets a new
FlowSkipStrategy , the current strategy is returned. |
void |
setFlowStepStrategy(FlowStepStrategy flowStepStrategy)
Sets a default
FlowStepStrategy instance. |
void |
setSubmitPriority(int submitPriority)
Method setSubmitPriority sets the submitPriority of this Flow object.
|
void |
start()
Method start begins the execution of this Flow instance.
|
boolean |
stepsAreLocal()
Method jobsAreLocal returns true if all jobs are executed in-process as a single map and reduce task.
|
void |
stop()
Method stop stops all running jobs, killing any currently executing.
|
void |
writeDOT(java.lang.String filename)
Method writeDOT writes this Flow instance to the given filename as a DOT file for import into a graphics package.
|
void |
writeStepsDOT(java.lang.String filename)
Method writeStepsDOT writes this Flow step graph to the given filename as a DOT file for import into a graphics package.
|
getSpawnStrategy, getStats, setSpawnStrategy
static final java.lang.String CASCADING_FLOW_ID
java.lang.String getName()
getName
in interface UnitOfWork<FlowStats>
void prepare()
Cascade
to notify the given Flow it should initialize or clear any resources
necessary for start()
to be called successfully.
Specifically, this implementation calls BaseFlow.deleteSinksIfNotUpdate()
&& BaseFlow.deleteTrapsIfNotUpdate()
.prepare
in interface UnitOfWork<FlowStats>
java.io.IOException
- whenvoid start()
complete()
to block until this Flow completes.start
in interface UnitOfWork<FlowStats>
void stop()
stop
in interface UnitOfWork<FlowStats>
void complete()
complete
in interface UnitOfWork<FlowStats>
void cleanup()
cleanup
in interface UnitOfWork<FlowStats>
PlannerInfo getPlannerInfo()
PlatformInfo getPlatformInfo()
Config getConfig()
FlowConnector
for setting
default properties visible to children. Or see FlowStepStrategy
for setting properties on
individual steps before they are executed.Config getConfigCopy()
java.util.Map<java.lang.Object,java.lang.Object> getConfigAsProperties()
Map
of
key value pairs.java.lang.String getProperty(java.lang.String key)
key
- of type Stringjava.lang.String getID()
getID
in interface UnitOfWork<FlowStats>
java.util.Map<java.lang.String,java.lang.String> getFlowDescriptor()
FlowDef.addDescription(String, String)
to set values on a given Flow.
Flow descriptions provide meta-data to monitoring systems describing the workload a given Flow represents.
For known description types, see FlowDescriptors
.java.lang.String getTags()
getTags
in interface UnitOfWork<FlowStats>
int getSubmitPriority()
void setSubmitPriority(int submitPriority)
submitPriority
- the submitPriority of this FlowStep object.FlowProcess<Config> getFlowProcess()
FlowStats getFlowStats()
boolean hasListeners()
FlowListener
instances have been registered.void addListener(FlowListener flowListener)
flowListener
- of type FlowListenerboolean removeListener(FlowListener flowListener)
flowListener
- of type FlowListenerboolean hasStepListeners()
FlowStepListener
instances have been registered
with any of the FlowStep
s belonging to this instancevoid addStepListener(FlowStepListener flowStepListener)
flowStepListener
- of type addStepListenerboolean removeStepListener(FlowStepListener flowStepListener)
flowStepListener
- of type FlowStepListenerFlowStep
belonging to this instancejava.util.Map<java.lang.String,Tap> getSources()
java.util.List<java.lang.String> getSourceNames()
java.util.Collection<Tap> getSourcesCollection()
Collection
of source Tap
s for this Flow object.java.util.Map<java.lang.String,Tap> getSinks()
java.util.List<java.lang.String> getSinkNames()
java.util.Collection<Tap> getSinksCollection()
Collection
of sink Tap
s for this Flow object.Tap getSink()
java.util.Map<java.lang.String,Tap> getTraps()
java.util.List<java.lang.String> getTrapNames()
java.util.Collection<Tap> getTrapsCollection()
Collection
of trap Tap
s for this Flow object.java.util.Map<java.lang.String,Tap> getCheckpoints()
java.util.List<java.lang.String> getCheckpointNames()
java.util.Collection<Tap> getCheckpointsCollection()
Collection
of checkpoint Tap
s for this Flow object.FlowSkipStrategy getFlowSkipStrategy()
FlowSkipStrategy
used by this Flow.FlowSkipStrategy setFlowSkipStrategy(FlowSkipStrategy flowSkipStrategy)
FlowSkipStrategy
, the current strategy is returned.
FlowSkipStrategy instances define when a Flow instance should be skipped. The default strategy is FlowSkipIfSinkNotStale
.
An alternative strategy would be FlowSkipIfSinkExists
.
A FlowSkipStrategy will not be consulted when executing a Flow directly through start()
or complete()
. Only
when the Flow is executed through a Cascade
instance.flowSkipStrategy
- of type FlowSkipStrategyboolean isSkipFlow() throws java.io.IOException
Cascade
should skip this Flow instance. True is returned
if the current FlowSkipStrategy
returns true.java.io.IOException
- whenboolean areSinksStale() throws java.io.IOException
Tap.isReplace()
returns true.java.io.IOException
- whenboolean areSourcesNewer(long sinkModified) throws java.io.IOException
sinkModified
- of type longjava.io.IOException
- whenlong getSinkModified() throws java.io.IOException
Tap
managed by this Flow instance.
If zero (0) is returned, at least one of the sink resources does not exist. If minus one (-1) is returned,
atleast one of the sinks are marked for delete (returns true
).java.io.IOException
- whenFlowStepStrategy getFlowStepStrategy()
FlowStepStrategy
instance.void setFlowStepStrategy(FlowStepStrategy flowStepStrategy)
FlowStepStrategy
instance.
Use a FlowStepStrategy to change FlowStep
configuration properties
before the properties are submitted to the underlying platform for the step
unit of work.flowStepStrategy
- The FlowStepStrategy to use.java.util.List<FlowStep<Config>> getFlowSteps()
TupleEntryIterator openSource() throws java.io.IOException
java.io.IOException
- whenTupleEntryIterator openSource(java.lang.String name) throws java.io.IOException
name
- of type Stringjava.io.IOException
- whenTupleEntryIterator openSink() throws java.io.IOException
java.io.IOException
- whenTupleEntryIterator openSink(java.lang.String name) throws java.io.IOException
name
- of type Stringjava.io.IOException
- whenTupleEntryIterator openTrap() throws java.io.IOException
java.io.IOException
- whenTupleEntryIterator openTrap(java.lang.String name) throws java.io.IOException
name
- of type Stringjava.io.IOException
- whenboolean resourceExists(Tap tap) throws java.io.IOException
tap
- of type Tapjava.io.IOException
- whenTupleEntryIterator openTapForRead(Tap tap) throws java.io.IOException
TupleEntryIterator
for the given Tap instance.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.tap
- of type Tapjava.io.IOException
- when there is an error opening the resourceTupleEntryCollector openTapForWrite(Tap tap) throws java.io.IOException
tap
- of type Tapjava.io.IOException
- when there is an error opening the resourcevoid writeDOT(java.lang.String filename)
filename
- of type Stringvoid writeStepsDOT(java.lang.String filename)
filename
- of type Stringjava.lang.String getCascadeID()
java.lang.String getRunID()
boolean stepsAreLocal()
boolean isStopJobsOnExit()
true
.Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.