cascading.pipe
Class Checkpoint

java.lang.Object
  extended by cascading.pipe.Pipe
      extended by cascading.pipe.Checkpoint
All Implemented Interfaces:
FlowElement, Traceable, Serializable

public class Checkpoint
extends Pipe

The Checkpoint pipe, if supported by the current planner, will force data to be persisted at the point in the tuple stream an instance of Checkpoint is inserted into the pipe assembly.

If a checkpoint Tap is added to the FlowDef via the FlowDef.addCheckpoint(Checkpoint, cascading.tap.Tap) method, that Tap instance will be used to capture the intermediate result sets.

It is required that any Scheme used as a checkpoint must source Fields.UNKNOWN and sink Fields.ALL.

If used with a TextDelimited Scheme class and the hasHeader value is true, a header with the resolved field names will be written to the file.

This is especially useful for debugging complex flows.

For the HadoopFlowConnector and Hadoop platform, a Checkpoint will force a new MapReduce job (HadoopFlowStep into the Flow plan.

This can be important when used in conjunction with a HashJoin where all the operations upstream from the HashJoin significantly filter out data allowing it to fit in memory.

See Also:
Serialized Form

Field Summary
 
Fields inherited from class cascading.pipe.Pipe
configDef, name, parent, previous, stepConfigDef
 
Constructor Summary
Checkpoint(Pipe previous)
          Constructor Checkpoint creates a new Checkpoint pipe which inherits the name of its previous pipe.
Checkpoint(String name, Pipe previous)
          Constructor Checkpoint creates a new Checkpoint pipe with the given name.
 
Method Summary
 
Methods inherited from class cascading.pipe.Pipe
equals, getConfigDef, getHeads, getName, getParent, getPrevious, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasStepConfigDef, id, isEquivalentTo, named, names, outgoingScopeFor, pipes, print, printInternal, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, setParent, toString
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Constructor Detail

Checkpoint

public Checkpoint(Pipe previous)
Constructor Checkpoint creates a new Checkpoint pipe which inherits the name of its previous pipe.

Parameters:
previous - of type Pipe

Checkpoint

public Checkpoint(String name,
                  Pipe previous)
Constructor Checkpoint creates a new Checkpoint pipe with the given name.

Parameters:
previous - of type Pipe


Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.