public class Checkpoint extends Pipe
Tap
is added to the FlowDef
via the
FlowDef.addCheckpoint(Checkpoint, cascading.tap.Tap)
method, that Tap instance
will be used to capture the intermediate result sets.
It is required that any Scheme used as a checkpoint must source Fields.UNKNOWN
and
sink Fields.ALL
.
If used with a TextDelimited
Scheme
class and
the hasHeader
value is true
, a header with the resolved field names will be written to the file.
This is especially useful for debugging complex flows.
For the HadoopFlowConnector
and Hadoop platform, a Checkpoint will force a new
MapReduce job (HadoopFlowStep
into the Flow
plan.
This can be important when used in conjunction with a HashJoin
where all the operations upstream
from the HashJoin significantly filter out data allowing it to fit in memory.configDef, name, nodeConfigDef, parent, previous, stepConfigDef
Constructor and Description |
---|
Checkpoint(Pipe previous)
Constructor Checkpoint creates a new Checkpoint pipe which inherits the name of its previous pipe.
|
Checkpoint(String name,
Pipe previous)
Constructor Checkpoint creates a new Checkpoint pipe with the given name.
|
equals, getConfigDef, getHeads, getName, getNodeConfigDef, getParent, getPrevious, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isEquivalentTo, named, names, outgoingScopeFor, pipes, print, printInternal, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, setParent, toString
public Checkpoint(Pipe previous)
previous
- of type Pipepublic Checkpoint(String name, Pipe previous)
previous
- of type PipeCopyright © 2007-2015 Concurrent, Inc. All Rights Reserved.