public class Pipe extends java.lang.Object implements ScopedElement, FlowElement, java.io.Serializable, Traceable
Each
, Every
, GroupBy
,
CoGroup
, Merge
, HashJoin
, and SubAssembly
.
Pipes are chained together through their constructors.
To effect a split in the pipe,
simply pass a Pipe instance to two or more constructors of subsequent Pipe instances.
A join can be achieved by passing two or more Pipe instances to a CoGroup
or HashJoin
pipe.
A merge can be achieved by passing two or more Pipe instances to a GroupBy
or Merge
pipe.Each
,
Every
,
GroupBy
,
Merge
,
CoGroup
,
HashJoin
,
SubAssembly
,
Serialized FormModifier and Type | Field and Description |
---|---|
protected ConfigDef |
configDef |
protected java.lang.String |
name
Field name
|
protected ConfigDef |
nodeConfigDef |
protected Pipe |
parent
Field parent
|
protected Pipe |
previous
Field previous
|
protected ConfigDef |
stepConfigDef |
Modifier | Constructor and Description |
---|---|
protected |
Pipe() |
protected |
Pipe(Pipe previous) |
|
Pipe(java.lang.String name)
Constructor Pipe creates a new Pipe instance with the given name.
|
|
Pipe(java.lang.String name,
Pipe previous)
Constructor Pipe creates a new Pipe instance with the given name and previous Pipe instance.
|
Modifier and Type | Method and Description |
---|---|
boolean |
equals(java.lang.Object object) |
ConfigDef |
getConfigDef()
Returns a
ConfigDef instance that allows for local properties to be set and made available via
a resulting FlowProcess instance when the pipe is invoked. |
Pipe[] |
getHeads()
Method getHeads returns the first Pipe instances in this pipe assembly.
|
java.lang.String |
getName()
Get the name of this pipe.
|
ConfigDef |
getNodeConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the pipe is invoked. |
Pipe |
getParent()
Returns the enclosing parent Pipe instance, if any.
|
Pipe[] |
getPrevious()
Get all the upstream pipes this pipe is connected to.
|
ConfigDef |
getStepConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the pipe is invoked. |
java.lang.String |
getTrace()
Method getTrace returns a String that pinpoints the caller that created this instance.
|
boolean |
hasConfigDef()
Returns
true if there are properties in the configDef instance. |
int |
hashCode() |
boolean |
hasNodeConfigDef()
Returns
true if there are properties in the nodeConfigDef instance. |
boolean |
hasStepConfigDef()
Returns
true if there are properties in the stepConfigDef instance. |
static java.lang.String |
id(Pipe pipe) |
static Pipe[] |
named(java.lang.String name,
Pipe... tails) |
static java.lang.String[] |
names(Pipe... tails)
Convenience method for finding all Pipe names in an assembly.
|
Scope |
outgoingScopeFor(java.util.Set<Scope> incomingScopes)
Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
|
static Pipe[] |
pipes(Pipe... pipes)
Convenience method to create an array of Pipe instances.
|
java.lang.String |
print(Scope scope)
Method print is used internally.
|
protected void |
printInternal(java.lang.StringBuffer buffer,
Scope scope) |
Fields |
resolveIncomingOperationArgumentFields(Scope incomingScope)
Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing Operation arguments.
|
Fields |
resolveIncomingOperationPassThroughFields(Scope incomingScope)
Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing the Pipe outgoing tuple.
|
protected void |
setParent(Pipe parent) |
java.lang.String |
toString() |
protected java.lang.String name
protected ConfigDef stepConfigDef
protected ConfigDef nodeConfigDef
protected Pipe()
@ConstructorProperties(value="name") public Pipe(java.lang.String name)
name
- name for this branch of Pipes@ConstructorProperties(value={"name","previous"}) public Pipe(java.lang.String name, Pipe previous)
name
- name for this branch of Pipesprevious
- previous Pipe to receive input Tuples frompublic static Pipe[] pipes(Pipe... pipes)
pipes
- vararg list of pipespublic static java.lang.String[] names(Pipe... tails)
tails
- vararg list of all tails in given assemblypublic java.lang.String getName()
public Pipe[] getPrevious()
public Pipe getParent()
SubAssembly
that wraps
this instance.public ConfigDef getConfigDef()
ConfigDef
instance that allows for local properties to be set and made available via
a resulting FlowProcess
instance when the pipe is invoked.
Any properties set on the configDef will not show up in any Flow
or
FlowStep
process level configuration, but will override any of those values as seen by the
current Pipe instance.getConfigDef
in interface ScopedElement
public boolean hasConfigDef()
true
if there are properties in the configDef instance.hasConfigDef
in interface ScopedElement
public ConfigDef getNodeConfigDef()
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the pipe is invoked.
Any properties set on the nodeConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowNode
(in Apache Tez the Vertex configuration). Any value set in the
nodeConfigDef will be overridden by the pipe local #getConfigDef
instance.
Use this method to tweak properties in the process node this pipe instance is planned into. In the case of the
Apache Tez platform, when set on a GroupBy
instance, the number of gather partitions can be modified.
In the case of any Pipe that spans FlowNode boundaries, like GroupBy and CoGroup may on some platforms,
any ConfigDef properties will be applied to the downstream FlowNode. That is, if a GroupBy is the source
to a node, any node ConfigDef properties will be applied. If the GroupBy encountered when applying properties
is on the sink side of a node, the properties will be ignored.getNodeConfigDef
in interface ScopedElement
public boolean hasNodeConfigDef()
true
if there are properties in the nodeConfigDef instance.hasNodeConfigDef
in interface ScopedElement
public ConfigDef getStepConfigDef()
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the pipe is invoked.
Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowStep
(in Hadoop the MapReduce jobconf). Any value set in the
stepConfigDef will be overridden by the pipe local #getConfigDef
instance.
Use this method to tweak properties in the process step this pipe instance is planned into. In the case of the
Hadoop platform, when set on a GroupBy
instance, the number of reducers can be modified.getStepConfigDef
in interface ScopedElement
public boolean hasStepConfigDef()
true
if there are properties in the stepConfigDef instance.hasStepConfigDef
in interface ScopedElement
public Pipe[] getHeads()
public Scope outgoingScopeFor(java.util.Set<Scope> incomingScopes)
ScopedElement
outgoingScopeFor
in interface ScopedElement
incomingScopes
- of type Setpublic Fields resolveIncomingOperationArgumentFields(Scope incomingScope)
ScopedElement
resolveIncomingOperationArgumentFields
in interface ScopedElement
incomingScope
- of type Scopepublic Fields resolveIncomingOperationPassThroughFields(Scope incomingScope)
ScopedElement
resolveIncomingOperationPassThroughFields
in interface ScopedElement
incomingScope
- of type Scopepublic java.lang.String getTrace()
Traceable
public java.lang.String toString()
toString
in class java.lang.Object
public boolean equals(java.lang.Object object)
equals
in class java.lang.Object
public int hashCode()
hashCode
in class java.lang.Object
public java.lang.String print(Scope scope)
scope
- of type Scopeprotected void printInternal(java.lang.StringBuffer buffer, Scope scope)
Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.