public interface Cascade extends UnitOfWork<CascadeStats>
Flow
instances that share or depend on equivalent Tap
instances and are executed as
a single group. The most common case is where one Flow instance depends on a Tap created by a second Flow instance. This
dependency chain can continue as practical.
Note Flow instances that have no shared dependencies will be executed in parallel.
Additionally, a Cascade allows for incremental builds of complex data processing processes. If a given source Tap
is newer than
a subsequent sink Tap
in the assembly, the connecting Flow
(s) will be executed
when the Cascade executed. If all the targets (sinks) are up to date, the Cascade exits immediately and does nothing.
The concept of 'stale' is pluggable, see the FlowSkipStrategy
class.
When a Cascade starts up, if first verifies which Flow instances have stale sinks, if the sinks are not stale, the
method BaseFlow.deleteSinksIfNotUpdate()
is called. Before appends/updates were supported (logically)
the Cascade deleted all the sinks in a Flow.
The new consequence of this is if the Cascade fails, but does complete a Flow that appended or updated data, re-running
the Cascade (and the successful append/update Flow) will re-update data to the source. Some systems may be idempotent and
may not have any side-effects. So plan accordingly.
Use the CascadeListener
to receive any events on the life-cycle of the Cascade as it executes. Any
Tap
instances owned by managed Flows also implementing CascadeListener will automatically be added to the
set of listeners.CascadeListener
,
Flow
,
FlowSkipStrategy
Modifier and Type | Method and Description |
---|---|
void |
addListener(CascadeListener cascadeListener) |
void |
complete()
Method complete begins the current Cascade process if method
start() was not previously called. |
List<Flow> |
findFlows(String regex)
Method findFlows returns a List of flows whose names match the given regex pattern.
|
Collection<Flow> |
findFlowsSinkingTo(String identifier)
Method findFlowsSinkingTo returns all Flow instances that writes to a sink with the given identifier.
|
Collection<Flow> |
findFlowsSourcingFrom(String identifier)
Method findFlowsSourcingFrom returns all Flow instances that reads from a source with the given identifier.
|
Collection<Tap> |
getAllTaps()
Method getAllTaps returns all source, sink, and checkpoint Tap instances associated with the managed
Flow instances in this Cascade instance.
|
CascadeStats |
getCascadeStats()
Method getCascadeStats returns the cascadeStats of this Cascade object.
|
Collection<Tap> |
getCheckpointsTaps()
Method getCheckpointTaps returns all checkpoint Tap instances from all the Flow instances in this Cascade instance.
|
List<Flow> |
getFlows()
Method getFlows returns the flows managed by this Cascade object.
|
FlowSkipStrategy |
getFlowSkipStrategy()
Method getFlowSkipStrategy returns the current
FlowSkipStrategy used by this Flow. |
Collection<Flow> |
getHeadFlows()
Method getHeadFlows returns all Flow instances that are at the "head" of the flow graph.
|
Collection<Flow> |
getIntermediateFlows()
Method getIntermediateFlows returns all Flow instances that are neither at the "tail" or "tail" of the flow graph.
|
Collection<Tap> |
getIntermediateTaps()
Method getIntermediateTaps returns all Tap instances that are neither at the source or sink of the flow graph.
|
Collection<Flow> |
getPredecessorFlows(Flow flow)
Method getPredecessorFlows returns a Collection of all the Flow instances that will be
executed before the given Flow instance.
|
Collection<Tap> |
getSinkTaps()
Method getSinkTaps returns all sink Tap instances in this Cascade instance.
|
Collection<Tap> |
getSourceTaps()
Method getSourceTaps returns all source Tap instances in this Cascade instance.
|
Collection<Flow> |
getSuccessorFlows(Flow flow)
Method getSuccessorFlows returns a Collection of all the Flow instances that will be
executed after the given Flow instance.
|
Collection<Flow> |
getTailFlows()
Method getTailFlows returns all Flow instances that are at the "tail" of the flow graph.
|
boolean |
hasListeners() |
boolean |
removeListener(CascadeListener flowListener) |
FlowSkipStrategy |
setFlowSkipStrategy(FlowSkipStrategy flowSkipStrategy)
Method setFlowSkipStrategy sets a new
FlowSkipStrategy , the current strategy, if any, is returned. |
void |
start()
Method start begins the current Cascade process.
|
void |
stop() |
void |
writeDOT(String filename)
Method writeDOT writes this element graph to a DOT file for easy visualization and debugging.
|
cleanup, getID, getName, getSpawnStrategy, getStats, getTags, prepare, setSpawnStrategy
boolean hasListeners()
void addListener(CascadeListener cascadeListener)
boolean removeListener(CascadeListener flowListener)
CascadeStats getCascadeStats()
List<Flow> getFlows()
Flow
instances
will be in topological order.List<Flow> findFlows(String regex)
regex
- of type StringCollection<Flow> getHeadFlows()
Collection<Flow> getTailFlows()
Collection<Flow> getIntermediateFlows()
Collection<Tap> getSourceTaps()
CompositeTap
instances are unwound if addressed directly by a managed Flow instance.Collection<Tap> getSinkTaps()
CompositeTap
instances are unwound if addressed directly by a managed Flow instance.
This method will return checkpoint Taps managed by Flow instances if not used as a source by other Flow instances.Collection<Tap> getCheckpointsTaps()
Collection<Tap> getIntermediateTaps()
Collection<Tap> getAllTaps()
Collection<Flow> getSuccessorFlows(Flow flow)
flow
- of type FlowCollection<Flow> getPredecessorFlows(Flow flow)
flow
- of type FlowCollection<Flow> findFlowsSourcingFrom(String identifier)
identifier
- of type StringCollection<Flow> findFlowsSinkingTo(String identifier)
identifier
- of type StringFlowSkipStrategy getFlowSkipStrategy()
FlowSkipStrategy
used by this Flow.FlowSkipStrategy setFlowSkipStrategy(FlowSkipStrategy flowSkipStrategy)
FlowSkipStrategy
, the current strategy, if any, is returned.
If a strategy is given, it will be used as the strategy for all BaseFlow
instances managed by this Cascade instance.
To revert back to consulting the strategies associated with each Flow instance, re-set this value to null
, its
default value.
FlowSkipStrategy instances define when a Flow instance should be skipped. The default strategy is FlowSkipIfSinkNotStale
and is inherited from the Flow instance in question. An alternative strategy would be FlowSkipIfSinkExists
.
A FlowSkipStrategy will not be consulted when executing a Flow directly through start()
flowSkipStrategy
- of type FlowSkipStrategyvoid start()
complete()
to block
until the Cascade completes.start
in interface UnitOfWork<CascadeStats>
void complete()
start()
was not previously called. This method
blocks until the process completes.complete
in interface UnitOfWork<CascadeStats>
RuntimeException
- wrapping any exception thrown internally.void stop()
stop
in interface UnitOfWork<CascadeStats>
Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.