public class DecoratorTap<MetaInfo,Config,Input,Output> extends Tap<Config,Input,Output>
Tap
instance, delegating all calls to the original.
It also provides an additional generic field that may hold any custom type, this allows implementations
to attach any meta-info to the tap being decorated.
Further, sub-classes of DecoratorTap can be used to decorate any Tap instances associated or created for
use by the Checkpoint
Pipe
.
Sub-classing this is optional if the aim is to simply attach relevant meta-info for use by a given application.
In order to pass any meta-info to a management service via the Property
annotation, a sub-class of DecoratorTap must be provided.Modifier and Type | Field and Description |
---|---|
protected MetaInfo |
metaInfo |
protected Tap<Config,Input,Output> |
original |
Constructor and Description |
---|
DecoratorTap(MetaInfo metaInfo,
Tap<Config,Input,Output> original)
Creates a new Tap instance, wrapping the given Tap, and associates the given MetaInfo type with
the wrapped Tap instance.
|
DecoratorTap(Tap<Config,Input,Output> original)
Creates a new Tap instance, wrapping the given Tap, and associates the given MetaInfo type with
the wrapped Tap instance.
|
Modifier and Type | Method and Description |
---|---|
boolean |
commitResource(Config conf)
Method commitResource allows the underlying resource to be notified when all write processing is
successful so that any additional cleanup or processing may be completed.
|
boolean |
createResource(Config conf)
Method createResource creates the underlying resource.
|
boolean |
createResource(FlowProcess<? extends Config> flowProcess)
Method createResource creates the underlying resource.
|
boolean |
deleteResource(Config conf)
Method deleteResource deletes the resource represented by this instance.
|
boolean |
deleteResource(FlowProcess<? extends Config> flowProcess)
Method deleteResource deletes the resource represented by this instance.
|
void |
flowConfInit(Flow<Config> flow)
Method flowInit allows this Tap instance to initialize itself in context of the given
Flow instance. |
ConfigDef |
getConfigDef()
Returns a
ConfigDef instance that allows for local properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
java.lang.String |
getFullIdentifier(Config conf)
Method getFullIdentifier returns a fully qualified resource identifier.
|
java.lang.String |
getFullIdentifier(FlowProcess<? extends Config> flowProcess)
Method getFullIdentifier returns a fully qualified resource identifier.
|
java.lang.String |
getIdentifier()
Method getIdentifier returns a String representing the resource this Tap instance represents.
|
MetaInfo |
getMetaInfo() |
long |
getModifiedTime(Config conf)
Method getModifiedTime returns the date this resource was last modified.
|
long |
getModifiedTime(FlowProcess<? extends Config> flowProcess)
Method getModifiedTime returns the date this resource was last modified.
|
ConfigDef |
getNodeConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
Tap<Config,Input,Output> |
getOriginal() |
Scheme<Config,Input,Output,?,?> |
getScheme()
Method getScheme returns the scheme of this Tap object.
|
Fields |
getSinkFields()
Method getSinkFields returns the sinkFields of this Tap object.
|
SinkMode |
getSinkMode()
Method getSinkMode returns the
SinkMode }of this Tap object. |
Fields |
getSourceFields()
Method getSourceFields returns the sourceFields of this Tap object.
|
ConfigDef |
getStepConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
java.lang.String |
getTrace()
Method getTrace returns a String that pinpoints the caller that created this instance.
|
boolean |
hasConfigDef()
Returns
true if there are properties in the configDef instance. |
boolean |
hasNodeConfigDef()
Returns
true if there are properties in the nodeConfigDef instance. |
boolean |
hasStepConfigDef()
Returns
true if there are properties in the stepConfigDef instance. |
boolean |
isKeep()
Method isKeep indicates whether the resource represented by this instance should be kept if it
already exists when the Flow is started.
|
boolean |
isReplace()
Method isReplace indicates whether the resource represented by this instance should be deleted if it
already exists when the Flow is started.
|
boolean |
isSink()
Method isSink returns true if this Tap instance can be used as a sink.
|
boolean |
isSource()
Method isSource returns true if this Tap instance can be used as a source.
|
boolean |
isTemporary()
Method isTemporary returns true if this Tap is temporary (used for intermediate results).
|
boolean |
isUpdate()
Method isUpdate indicates whether the resource represented by this instance should be updated if it already
exists.
|
TupleEntryIterator |
openForRead(FlowProcess<? extends Config> flowProcess)
Method openForRead opens the resource represented by this Tap instance for reading.
|
TupleEntryIterator |
openForRead(FlowProcess<? extends Config> flowProcess,
Input input)
Method openForRead opens the resource represented by this Tap instance for reading.
|
TupleEntryCollector |
openForWrite(FlowProcess<? extends Config> flowProcess)
Method openForWrite opens the resource represented by this Tap instance for writing.
|
TupleEntryCollector |
openForWrite(FlowProcess<? extends Config> flowProcess,
Output output)
Method openForWrite opens the resource represented by this Tap instance for writing.
|
Scope |
outgoingScopeFor(java.util.Set<Scope> incomingScopes)
Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
|
boolean |
prepareResourceForRead(Config conf)
Method prepareResourceForRead allows the underlying resource to be notified when reading will begin.
|
boolean |
prepareResourceForWrite(Config conf)
Method prepareResourceForWrite allows the underlying resource to be notified when writing will begin.
|
void |
presentSinkFields(FlowProcess<? extends Config> flowProcess,
Fields fields) |
void |
presentSourceFields(FlowProcess<? extends Config> flowProcess,
Fields fields) |
Fields |
resolveIncomingOperationArgumentFields(Scope incomingScope)
Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing Operation arguments.
|
Fields |
resolveIncomingOperationPassThroughFields(Scope incomingScope)
Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing the Pipe outgoing tuple.
|
boolean |
resourceExists(Config conf)
Method resourceExists returns true if the path represented by this instance exists.
|
boolean |
resourceExists(FlowProcess<? extends Config> flowProcess)
Method resourceExists returns true if the path represented by this instance exists.
|
Fields |
retrieveSinkFields(FlowProcess<? extends Config> flowProcess)
A hook for allowing a Scheme to lazily retrieve its sink fields.
|
Fields |
retrieveSourceFields(FlowProcess<? extends Config> flowProcess)
A hook for allowing a Scheme to lazily retrieve its source fields.
|
boolean |
rollbackResource(Config conf)
Method rollbackResource allows the underlying resource to be notified when any write processing has failed or
was stopped so that any cleanup may be started.
|
protected void |
setMetaInfo(MetaInfo metaInfo) |
protected void |
setOriginal(Tap<Config,Input,Output> original) |
void |
sinkConfInit(FlowProcess<? extends Config> flowProcess,
Config conf)
Method sinkConfInit initializes this instance as a sink.
|
void |
sourceConfInit(FlowProcess<? extends Config> flowProcess,
Config conf)
Method sourceConfInit initializes this instance as a source.
|
java.lang.String |
toString() |
@ConstructorProperties(value={"metaInfo","original"}) public DecoratorTap(MetaInfo metaInfo, Tap<Config,Input,Output> original)
metaInfo
- meta-information about the current Taporiginal
- the decorated Tap instance@ConstructorProperties(value="original") public DecoratorTap(Tap<Config,Input,Output> original)
original
- the decorated Tap instancepublic MetaInfo getMetaInfo()
public Tap<Config,Input,Output> getOriginal()
protected void setOriginal(Tap<Config,Input,Output> original)
protected void setMetaInfo(MetaInfo metaInfo)
public Scheme<Config,Input,Output,?,?> getScheme()
Tap
public java.lang.String getTrace()
Traceable
public void flowConfInit(Flow<Config> flow)
Tap
Flow
instance.
This method is guaranteed to be called before the Flow is started and the
FlowListener.onStarting(cascading.flow.Flow)
event is fired.
This method will be called once per Flow, and before Tap.sourceConfInit(cascading.flow.FlowProcess, Object)
and
Tap.sinkConfInit(cascading.flow.FlowProcess, Object)
methods.flowConfInit
in class Tap<Config,Input,Output>
flow
- of type Flowpublic void sourceConfInit(FlowProcess<? extends Config> flowProcess, Config conf)
Tap
Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method.sourceConfInit
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessconf
- of type Configpublic void sinkConfInit(FlowProcess<? extends Config> flowProcess, Config conf)
Tap
Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
Note this method will be called in context of this Tap being used as a traditional 'sink' and as a 'trap'.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method. If this Tap instance returns true for
Tap.isReplace()
, then Tap.deleteResource(Object)
will be called by the parent Flow.sinkConfInit
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessconf
- of type Configpublic java.lang.String getIdentifier()
Tap
getIdentifier
in class Tap<Config,Input,Output>
public Fields getSourceFields()
Tap
getSourceFields
in class Tap<Config,Input,Output>
public Fields getSinkFields()
Tap
getSinkFields
in class Tap<Config,Input,Output>
public TupleEntryIterator openForRead(FlowProcess<? extends Config> flowProcess, Input input) throws java.io.IOException
Tap
input
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
input type and instantiate it before calling super.openForRead()
.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.openForRead
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessinput
- of type Inputjava.io.IOException
- when the resource cannot be openedpublic TupleEntryIterator openForRead(FlowProcess<? extends Config> flowProcess) throws java.io.IOException
Tap
TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.openForRead
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessjava.io.IOException
- when the resource cannot be openedpublic TupleEntryCollector openForWrite(FlowProcess<? extends Config> flowProcess, Output output) throws java.io.IOException
Tap
SinkMode
setting. If SinkMode is
SinkMode.REPLACE
, this call may fail. See Tap.openForWrite(cascading.flow.FlowProcess)
.
output
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
output type and instantiate it before calling super.openForWrite()
.openForWrite
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessoutput
- of type Outputjava.io.IOException
- when the resource cannot be openedpublic TupleEntryCollector openForWrite(FlowProcess<? extends Config> flowProcess) throws java.io.IOException
Tap
SinkMode.REPLACE
settings. That is, if
SinkMode is set to SinkMode.REPLACE
the underlying resource will be deleted.
Note if SinkMode.UPDATE
is set, the resource will not be deleted.openForWrite
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessjava.io.IOException
- when the resource cannot be openedpublic Scope outgoingScopeFor(java.util.Set<Scope> incomingScopes)
ScopedElement
outgoingScopeFor
in interface ScopedElement
outgoingScopeFor
in class Tap<Config,Input,Output>
incomingScopes
- of type Setpublic Fields retrieveSourceFields(FlowProcess<? extends Config> flowProcess)
Tap
retrieveSourceFields
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcesspublic void presentSourceFields(FlowProcess<? extends Config> flowProcess, Fields fields)
presentSourceFields
in class Tap<Config,Input,Output>
public Fields retrieveSinkFields(FlowProcess<? extends Config> flowProcess)
Tap
retrieveSinkFields
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcesspublic void presentSinkFields(FlowProcess<? extends Config> flowProcess, Fields fields)
presentSinkFields
in class Tap<Config,Input,Output>
public Fields resolveIncomingOperationArgumentFields(Scope incomingScope)
ScopedElement
resolveIncomingOperationArgumentFields
in interface ScopedElement
resolveIncomingOperationArgumentFields
in class Tap<Config,Input,Output>
incomingScope
- of type Scopepublic Fields resolveIncomingOperationPassThroughFields(Scope incomingScope)
ScopedElement
resolveIncomingOperationPassThroughFields
in interface ScopedElement
resolveIncomingOperationPassThroughFields
in class Tap<Config,Input,Output>
incomingScope
- of type Scopepublic java.lang.String getFullIdentifier(FlowProcess<? extends Config> flowProcess)
Tap
getFullIdentifier
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcesspublic java.lang.String getFullIdentifier(Config conf)
Tap
getFullIdentifier
in class Tap<Config,Input,Output>
conf
- of type Configpublic boolean createResource(FlowProcess<? extends Config> flowProcess) throws java.io.IOException
Tap
createResource
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessjava.io.IOException
- when there is an error making directoriespublic boolean createResource(Config conf) throws java.io.IOException
Tap
createResource
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
- when there is an error making directoriespublic boolean deleteResource(FlowProcess<? extends Config> flowProcess) throws java.io.IOException
Tap
deleteResource
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessjava.io.IOException
- when the resource cannot be deletedpublic boolean deleteResource(Config conf) throws java.io.IOException
Tap
deleteResource
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
- when the resource cannot be deletedpublic boolean prepareResourceForRead(Config conf) throws java.io.IOException
Tap
false
, an exception will be thrown halting the current Flow.
In most cases, resource initialization should happen in the Tap.openForRead(FlowProcess, Object)
method.
This allows for initialization of cluster side resources, like a JDBC driver used to read data from a database,
that cannot be passed client to cluster.prepareResourceForRead
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
public boolean prepareResourceForWrite(Config conf) throws java.io.IOException
Tap
false
, an exception will be thrown halting the current Flow.
In most cases, resource initialization should happen in the Tap.openForWrite(FlowProcess, Object)
method.
This allows for initialization of cluster side resources, like a JDBC driver used to write data to a database,
that cannot be passed client to cluster.
In the above JDBC example, overriding this method will allow for testing for the existence of and/or creating
a remote table used by all individual cluster side tasks.prepareResourceForWrite
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
public boolean commitResource(Config conf) throws java.io.IOException
Tap
Tap.rollbackResource(Object)
to handle cleanup in the face of failures.
This method is invoked once client side and not in the cluster, if any.
If other sink Tap instance in a given Flow fail on commitResource after called on this instance,
rollbackResource will not be called.commitResource
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
public boolean rollbackResource(Config conf) throws java.io.IOException
Tap
Tap.commitResource(Object)
to handle cleanup when the write has successfully completed.
This method is invoked once client side and not in the cluster, if any.rollbackResource
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
public boolean resourceExists(FlowProcess<? extends Config> flowProcess) throws java.io.IOException
Tap
resourceExists
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessjava.io.IOException
- when the status cannot be determinedpublic boolean resourceExists(Config conf) throws java.io.IOException
Tap
resourceExists
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
- when the status cannot be determinedpublic long getModifiedTime(FlowProcess<? extends Config> flowProcess) throws java.io.IOException
Tap
getModifiedTime
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessjava.io.IOException
public long getModifiedTime(Config conf) throws java.io.IOException
Tap
getModifiedTime
in class Tap<Config,Input,Output>
conf
- of type Configjava.io.IOException
public SinkMode getSinkMode()
Tap
SinkMode
}of this Tap object.getSinkMode
in class Tap<Config,Input,Output>
public boolean isKeep()
Tap
public boolean isReplace()
Tap
public boolean isUpdate()
Tap
Tap.createResource(Object)
, when the Flow is started.public boolean isSink()
Tap
public boolean isSource()
Tap
public boolean isTemporary()
Tap
isTemporary
in class Tap<Config,Input,Output>
public ConfigDef getConfigDef()
Tap
ConfigDef
instance that allows for local properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the configDef will not show up in any Flow
or FlowStep
process
level configuration, but will override any of those values as seen by the current Tap instance method call where a
FlowProcess is provided except for the Tap.sourceConfInit(cascading.flow.FlowProcess, Object)
and
Tap.sinkConfInit(cascading.flow.FlowProcess, Object)
methods.
That is, the *confInit
methods are called before any ConfigDef is applied, so any values placed into
a ConfigDef instance will not be visible to them.getConfigDef
in interface ScopedElement
getConfigDef
in class Tap<Config,Input,Output>
public boolean hasConfigDef()
Tap
true
if there are properties in the configDef instance.hasConfigDef
in interface ScopedElement
hasConfigDef
in class Tap<Config,Input,Output>
public ConfigDef getNodeConfigDef()
Tap
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the nodeConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowNode
(in Apache Tez the Vertex configuration). Any value set in the
nodeConfigDef will be overridden by the pipe local #getConfigDef
instance.
Use this method to tweak properties in the process node this tap instance is planned into.getNodeConfigDef
in interface ScopedElement
getNodeConfigDef
in class Tap<Config,Input,Output>
public boolean hasNodeConfigDef()
Tap
true
if there are properties in the nodeConfigDef instance.hasNodeConfigDef
in interface ScopedElement
hasNodeConfigDef
in class Tap<Config,Input,Output>
public ConfigDef getStepConfigDef()
Tap
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowStep
(in Hadoop the MapReduce jobconf). Any value set in the
stepConfigDef will be overridden by the tap local #getConfigDef
instance.
Use this method to tweak properties in the process step this tap instance is planned into.
Note the *confInit
methods are called before any ConfigDef is applied, so any values placed into
a ConfigDef instance will not be visible to them.getStepConfigDef
in interface ScopedElement
getStepConfigDef
in class Tap<Config,Input,Output>
public boolean hasStepConfigDef()
Tap
true
if there are properties in the stepConfigDef instance.hasStepConfigDef
in interface ScopedElement
hasStepConfigDef
in class Tap<Config,Input,Output>
Copyright © 2007-2015 Xplenty, Inc. All Rights Reserved.