cascading.pipe

Class Checkpoint

java.lang.Object

cascading.pipe.Pipe

cascading.pipe.Checkpoint

All Implemented Interfaces:

FlowElement, ScopedElement, Traceable, java.io.Serializable

public class Checkpoint
extends Pipe

The Checkpoint pipe, if supported by the current planner, will force data to be persisted at the point in the tuple stream an instance of Checkpoint is inserted into the pipe assembly.

If a checkpoint Tap is added to the FlowDef via the FlowDef.addCheckpoint(Checkpoint, cascading.tap.Tap) method, that Tap instance will be used to capture the intermediate result sets.

It is required that any Scheme used as a checkpoint must source Fields.UNKNOWN and sink Fields.ALL.

If used with a TextDelimited Scheme class and the hasHeader value is true, a header with the resolved field names will be written to the file.

This is especially useful for debugging complex flows.

For the HadoopFlowConnector and Hadoop platform, a Checkpoint will force a new MapReduce job (HadoopFlowStep into the Flow plan.

This can be important when used in conjunction with a HashJoin where all the operations upstream from the HashJoin significantly filter out data allowing it to fit in memory.

See Also:

Serialized Form

Field Summary

Fields inherited from class cascading.pipe.Pipe

configDef, name, nodeConfigDef, parent, previous, stepConfigDef

Constructor Summary

Constructors

Constructor and Description
Checkpoint(Pipe previous) Constructor Checkpoint creates a new Checkpoint pipe which inherits the name of its previous pipe.
Checkpoint(java.lang.String name, Pipe previous) Constructor Checkpoint creates a new Checkpoint pipe with the given name.

Method Summary

Methods inherited from class cascading.pipe.Pipe

equals, getConfigDef, getHeads, getName, getNodeConfigDef, getParent, getPrevious, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, named, names, outgoingScopeFor, pipes, print, printInternal, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, setParent, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

Checkpoint

public Checkpoint(Pipe previous)

Constructor Checkpoint creates a new Checkpoint pipe which inherits the name of its previous pipe.

Parameters:

previous - of type Pipe

Checkpoint

public Checkpoint(java.lang.String name,
          Pipe previous)

Constructor Checkpoint creates a new Checkpoint pipe with the given name.

Parameters:

previous - of type Pipe