cascading.pipe
Class Pipe

java.lang.Object
  extended by cascading.pipe.Pipe
All Implemented Interfaces:
FlowElement, Traceable, Serializable
Direct Known Subclasses:
Checkpoint, ElementGraph.Extent, Operator, Splice, SubAssembly

public class Pipe
extends Object
implements FlowElement, Serializable, Traceable

Class Pipe is used to name branches in pipe assemblies, and as a base class for core processing model types, specifically Each, Every, GroupBy, CoGroup, Merge, HashJoin, and SubAssembly.

Pipes are chained together through their constructors.

To effect a split in the pipe, simply pass a Pipe instance to two or more constructors of subsequent Pipe instances.

A join can be achieved by passing two or more Pipe instances to a CoGroup or HashJoin pipe.

A merge can be achieved by passing two or more Pipe instances to a GroupBy or Merge pipe.

See Also:
Each, Every, GroupBy, Merge, CoGroup, HashJoin, SubAssembly, Serialized Form

Field Summary
protected  ConfigDef configDef
           
protected  String name
          Field name
protected  Pipe parent
          Field parent
protected  Pipe previous
          Field previous
protected  ConfigDef stepConfigDef
           
 
Constructor Summary
protected Pipe()
           
protected Pipe(Pipe previous)
           
  Pipe(String name)
          Constructor Pipe creates a new Pipe instance with the given name.
  Pipe(String name, Pipe previous)
          Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance.
 
Method Summary
 boolean equals(Object object)
           
 ConfigDef getConfigDef()
          Returns a ConfigDef instance that allows for local properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.
 Pipe[] getHeads()
          Method getHeads returns the first Pipe instances in this pipe assembly.
 String getName()
          Get the name of this pipe.
 Pipe getParent()
          Returns the enclosing parent Pipe instance, if any.
 Pipe[] getPrevious()
          Get all the upstream pipes this pipe is connected to.
 ConfigDef getStepConfigDef()
          Returns a ConfigDef instance that allows for process level properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.
 String getTrace()
          Method getTrace returns a String that pinpoints the caller that created this instance.
 boolean hasConfigDef()
          Returns true if there are properties in the configDef instance.
 int hashCode()
           
 boolean hasStepConfigDef()
          Returns true if there are properties in the processConfigDef instance.
static String id(Pipe pipe)
           
 boolean isEquivalentTo(FlowElement element)
           
static Pipe[] named(String name, Pipe... tails)
           
static String[] names(Pipe... tails)
          Convenience method for finding all Pipe names in an assembly.
 Scope outgoingScopeFor(Set<Scope> incomingScopes)
          Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
static Pipe[] pipes(Pipe... pipes)
          Convenience method to create an array of Pipe instances.
 String print(Scope scope)
          Method print is used internally.
protected  void printInternal(StringBuffer buffer, Scope scope)
           
 Fields resolveIncomingOperationArgumentFields(Scope incomingScope)
          Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing Operation arguments.
 Fields resolveIncomingOperationPassThroughFields(Scope incomingScope)
          Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing the Pipe outgoing tuple.
protected  void setParent(Pipe parent)
           
 String toString()
           
 
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
 

Field Detail

name

protected String name
Field name


previous

protected Pipe previous
Field previous


parent

protected Pipe parent
Field parent


configDef

protected ConfigDef configDef

stepConfigDef

protected ConfigDef stepConfigDef
Constructor Detail

Pipe

protected Pipe()

Pipe

@ConstructorProperties(value="previous")
protected Pipe(Pipe previous)

Pipe

@ConstructorProperties(value="name")
public Pipe(String name)
Constructor Pipe creates a new Pipe instance with the given name. This is useful as the 'start' or head of a pipe assembly.

Parameters:
name - name for this branch of Pipes

Pipe

@ConstructorProperties(value={"name","previous"})
public Pipe(String name,
                                       Pipe previous)
Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance. This is useful for naming a branch in a pipe assembly. Or renaming the branch mid-way down.

Parameters:
name - name for this branch of Pipes
previous - previous Pipe to receive input Tuples from
Method Detail

id

public static String id(Pipe pipe)

pipes

public static Pipe[] pipes(Pipe... pipes)
Convenience method to create an array of Pipe instances.

Parameters:
pipes - vararg list of pipes
Returns:
array of pipes

names

public static String[] names(Pipe... tails)
Convenience method for finding all Pipe names in an assembly.

Parameters:
tails - vararg list of all tails in given assembly
Returns:
array of Pipe names

named

public static Pipe[] named(String name,
                           Pipe... tails)

getName

public String getName()
Get the name of this pipe. Guaranteed non-null.

Returns:
String the name of this pipe

getPrevious

public Pipe[] getPrevious()
Get all the upstream pipes this pipe is connected to. This method will return the Pipe instances passed on the constructors as inputs to this Pipe instance.

Returns:
all the upstream pipes this pipe is connected to.

setParent

protected void setParent(Pipe parent)

getParent

public Pipe getParent()
Returns the enclosing parent Pipe instance, if any. A parent is typically a SubAssembly that wraps this instance.

Returns:
of type Pipe

getConfigDef

public ConfigDef getConfigDef()
Returns a ConfigDef instance that allows for local properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.

Any properties set on the configDef will not show up in any Flow or FlowStep process level configuration, but will override any of those values as seen by the current Pipe instance.

Specified by:
getConfigDef in interface FlowElement
Returns:
an instance of ConfigDef

hasConfigDef

public boolean hasConfigDef()
Returns true if there are properties in the configDef instance.

Specified by:
hasConfigDef in interface FlowElement
Returns:
true if there are configDef properties

getStepConfigDef

public ConfigDef getStepConfigDef()
Returns a ConfigDef instance that allows for process level properties to be set and made available via a resulting FlowProcess instance when the pipe is invoked.

Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in the current process FlowStep (in Hadoop the MapReduce jobconf). Any value set in the stepConfigDef will be overridden by the pipe local #getConfigDef instance.

Use this method to tweak properties in the process step this pipe instance is planned into. In the case of the Hadoop platform, when set on a GroupBy instance, the number of reducers can be modified.

Specified by:
getStepConfigDef in interface FlowElement
Returns:
an instance of ConfigDef

hasStepConfigDef

public boolean hasStepConfigDef()
Returns true if there are properties in the processConfigDef instance.

Specified by:
hasStepConfigDef in interface FlowElement
Returns:
true if there are processConfigDef properties

getHeads

public Pipe[] getHeads()
Method getHeads returns the first Pipe instances in this pipe assembly.

Returns:
the first (type Pipe[]) of this Pipe object.

outgoingScopeFor

public Scope outgoingScopeFor(Set<Scope> incomingScopes)
Description copied from interface: FlowElement
Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.

Specified by:
outgoingScopeFor in interface FlowElement
Parameters:
incomingScopes - of type Set
Returns:
Scope

resolveIncomingOperationArgumentFields

public Fields resolveIncomingOperationArgumentFields(Scope incomingScope)
Description copied from interface: FlowElement
Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing Operation arguments.

Specified by:
resolveIncomingOperationArgumentFields in interface FlowElement
Parameters:
incomingScope - of type Scope
Returns:
Fields

resolveIncomingOperationPassThroughFields

public Fields resolveIncomingOperationPassThroughFields(Scope incomingScope)
Description copied from interface: FlowElement
Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that are consumable by this FlowElement when preparing the Pipe outgoing tuple.

Specified by:
resolveIncomingOperationPassThroughFields in interface FlowElement
Parameters:
incomingScope - of type Scope
Returns:
Fields

getTrace

public String getTrace()
Description copied from interface: Traceable
Method getTrace returns a String that pinpoints the caller that created this instance.

Specified by:
getTrace in interface Traceable
Returns:
String

toString

public String toString()
Overrides:
toString in class Object

isEquivalentTo

public boolean isEquivalentTo(FlowElement element)
Specified by:
isEquivalentTo in interface FlowElement

equals

public boolean equals(Object object)
Overrides:
equals in class Object

hashCode

public int hashCode()
Overrides:
hashCode in class Object

print

public String print(Scope scope)
Method print is used internally.

Parameters:
scope - of type Scope
Returns:
String

printInternal

protected void printInternal(StringBuffer buffer,
                             Scope scope)


Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.