public class AdaptorTap<TConfig,TInput,TOutput,OConfig,OInput,OOutput> extends Tap<TConfig,TInput,TOutput>
Tap
instance, delegating all calls to the original. In addition, AdaptorTap
implementations must provide Function
implementations that will adapt one Config and FlowProcess
type from the current platform types to the underlying Tap platform types.Modifier and Type | Field and Description |
---|---|
protected java.util.function.Function<TConfig,OConfig> |
configProvider |
protected Tap<OConfig,OInput,OOutput> |
original |
protected java.util.function.Function<FlowProcess<? extends TConfig>,FlowProcess<? extends OConfig>> |
processProvider |
Constructor and Description |
---|
AdaptorTap(Tap<OConfig,OInput,OOutput> original,
java.util.function.Function<FlowProcess<? extends TConfig>,FlowProcess<? extends OConfig>> processProvider,
java.util.function.Function<TConfig,OConfig> configProvider) |
Modifier and Type | Method and Description |
---|---|
boolean |
commitResource(TConfig conf)
Method commitResource allows the underlying resource to be notified when all write processing is
successful so that any additional cleanup or processing may be completed.
|
boolean |
createResource(FlowProcess<? extends TConfig> flowProcess)
Method createResource creates the underlying resource.
|
boolean |
createResource(TConfig conf)
Method createResource creates the underlying resource.
|
boolean |
deleteResource(FlowProcess<? extends TConfig> flowProcess)
Method deleteResource deletes the resource represented by this instance.
|
boolean |
deleteResource(TConfig conf)
Method deleteResource deletes the resource represented by this instance.
|
boolean |
equals(java.lang.Object object) |
void |
flowConfInit(Flow<TConfig> flow)
Method flowInit allows this Tap instance to initialize itself in context of the given
Flow instance. |
ConfigDef |
getConfigDef()
Returns a
ConfigDef instance that allows for local properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
java.lang.String |
getFullIdentifier(FlowProcess<? extends TConfig> flowProcess)
Method getFullIdentifier returns a fully qualified resource identifier.
|
java.lang.String |
getFullIdentifier(TConfig conf)
Method getFullIdentifier returns a fully qualified resource identifier.
|
java.lang.String |
getIdentifier()
Method getIdentifier returns a String representing the resource this Tap instance represents.
|
long |
getModifiedTime(FlowProcess<? extends TConfig> flowProcess)
Method getModifiedTime returns the date this resource was last modified.
|
long |
getModifiedTime(TConfig conf)
Method getModifiedTime returns the date this resource was last modified.
|
ConfigDef |
getNodeConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
Tap<OConfig,OInput,OOutput> |
getOriginal() |
Scheme<TConfig,TInput,TOutput,?,?> |
getScheme()
Method getScheme returns the scheme of this Tap object.
|
Fields |
getSinkFields()
Method getSinkFields returns the sinkFields of this Tap object.
|
SinkMode |
getSinkMode()
Method getSinkMode returns the
SinkMode }of this Tap object. |
Fields |
getSourceFields()
Method getSourceFields returns the sourceFields of this Tap object.
|
ConfigDef |
getStepConfigDef()
Returns a
ConfigDef instance that allows for process level properties to be set and made available via
a resulting FlowProcess instance when the tap is invoked. |
java.lang.String |
getTrace()
Method getTrace returns a String that pinpoints the caller that created this instance.
|
boolean |
hasConfigDef()
Returns
true if there are properties in the configDef instance. |
int |
hashCode() |
boolean |
hasNodeConfigDef()
Returns
true if there are properties in the nodeConfigDef instance. |
boolean |
hasStepConfigDef()
Returns
true if there are properties in the stepConfigDef instance. |
boolean |
isKeep()
Method isKeep indicates whether the resource represented by this instance should be kept if it
already exists when the Flow is started.
|
boolean |
isReplace()
Method isReplace indicates whether the resource represented by this instance should be deleted if it
already exists when the Flow is started.
|
boolean |
isSink()
Method isSink returns true if this Tap instance can be used as a sink.
|
boolean |
isSource()
Method isSource returns true if this Tap instance can be used as a source.
|
boolean |
isTemporary()
Method isTemporary returns true if this Tap is temporary (used for intermediate results).
|
boolean |
isUpdate()
Method isUpdate indicates whether the resource represented by this instance should be updated if it already
exists.
|
TupleEntryIterator |
openForRead(FlowProcess<? extends TConfig> flowProcess)
Method openForRead opens the resource represented by this Tap instance for reading.
|
TupleEntryIterator |
openForRead(FlowProcess<? extends TConfig> flowProcess,
TInput input)
Method openForRead opens the resource represented by this Tap instance for reading.
|
TupleEntryCollector |
openForWrite(FlowProcess<? extends TConfig> flowProcess)
Method openForWrite opens the resource represented by this Tap instance for writing.
|
TupleEntryCollector |
openForWrite(FlowProcess<? extends TConfig> flowProcess,
TOutput output)
Method openForWrite opens the resource represented by this Tap instance for writing.
|
Scope |
outgoingScopeFor(java.util.Set<Scope> incomingScopes)
Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
|
boolean |
prepareResourceForRead(TConfig conf)
Method prepareResourceForRead allows the underlying resource to be notified when reading will begin.
|
boolean |
prepareResourceForWrite(TConfig conf)
Method prepareResourceForWrite allows the underlying resource to be notified when writing will begin.
|
void |
presentSinkFields(FlowProcess<? extends TConfig> flowProcess,
Fields fields) |
void |
presentSourceFields(FlowProcess<? extends TConfig> flowProcess,
Fields fields) |
Fields |
resolveIncomingOperationArgumentFields(Scope incomingScope)
Method resolveIncomingOperationArgumentFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing Operation arguments.
|
Fields |
resolveIncomingOperationPassThroughFields(Scope incomingScope)
Method resolveIncomingOperationPassThroughFields returns the Fields outgoing from the previous FlowElement that
are consumable by this FlowElement when preparing the Pipe outgoing tuple.
|
boolean |
resourceExists(FlowProcess<? extends TConfig> flowProcess)
Method resourceExists returns true if the path represented by this instance exists.
|
boolean |
resourceExists(TConfig conf)
Method resourceExists returns true if the path represented by this instance exists.
|
Fields |
retrieveSinkFields(FlowProcess<? extends TConfig> flowProcess)
A hook for allowing a Scheme to lazily retrieve its sink fields.
|
Fields |
retrieveSourceFields(FlowProcess<? extends TConfig> flowProcess)
A hook for allowing a Scheme to lazily retrieve its source fields.
|
boolean |
rollbackResource(TConfig conf)
Method rollbackResource allows the underlying resource to be notified when any write processing has failed or
was stopped so that any cleanup may be started.
|
protected void |
setOriginal(Tap<OConfig,OInput,OOutput> original) |
void |
sinkConfInit(FlowProcess<? extends TConfig> flowProcess,
TConfig conf)
Method sinkConfInit initializes this instance as a sink.
|
void |
sourceConfInit(FlowProcess<? extends TConfig> flowProcess,
TConfig conf)
Method sourceConfInit initializes this instance as a source.
|
java.lang.String |
toString() |
entryStream, entryStream, entryStreamCopy, entryStreamCopy, id, openForReadUnchecked, setScheme, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy
protected java.util.function.Function<FlowProcess<? extends TConfig>,FlowProcess<? extends OConfig>> processProvider
protected java.util.function.Function<TConfig,OConfig> configProvider
@ConstructorProperties(value={"original","processProvider","configProvider"}) public AdaptorTap(Tap<OConfig,OInput,OOutput> original, java.util.function.Function<FlowProcess<? extends TConfig>,FlowProcess<? extends OConfig>> processProvider, java.util.function.Function<TConfig,OConfig> configProvider)
public Tap<OConfig,OInput,OOutput> getOriginal()
protected void setOriginal(Tap<OConfig,OInput,OOutput> original)
public Scheme<TConfig,TInput,TOutput,?,?> getScheme()
Tap
public java.lang.String getTrace()
Traceable
public void flowConfInit(Flow<TConfig> flow)
Tap
Flow
instance.
This method is guaranteed to be called before the Flow is started and the
FlowListener.onStarting(cascading.flow.Flow)
event is fired.
This method will be called once per Flow, and before Tap.sourceConfInit(cascading.flow.FlowProcess, Object)
and
Tap.sinkConfInit(cascading.flow.FlowProcess, Object)
methods.
flowConfInit
in class Tap<TConfig,TInput,TOutput>
flow
- of type Flowpublic void sourceConfInit(FlowProcess<? extends TConfig> flowProcess, TConfig conf)
Tap
This method maybe called more than once if this Tap instance is used outside the scope of a Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method.
sourceConfInit
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessconf
- of type Configpublic void sinkConfInit(FlowProcess<? extends TConfig> flowProcess, TConfig conf)
Tap
This method maybe called more than once if this Tap instance is used outside the scope of a Flow
instance or if it participates in multiple times in a given Flow or across different Flows in
a Cascade
.
Note this method will be called in context of this Tap being used as a traditional 'sink' and as a 'trap'.
In the context of a Flow, it will be called after
FlowListener.onStarting(cascading.flow.Flow)
Note that no resources or services should be modified by this method. If this Tap instance returns true for
Tap.isReplace()
, then Tap.deleteResource(Object)
will be called by the parent Flow.
sinkConfInit
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessconf
- of type Configpublic java.lang.String getIdentifier()
Tap
Often, if the tap accesses a filesystem, the identifier is nothing more than the path to the file or directory. In other cases it may be a an URL or URI representing a connection string or remote resource.
Any two Tap instances having the same value for the identifier are considered equal.
getIdentifier
in class Tap<TConfig,TInput,TOutput>
public Fields getSourceFields()
Tap
getSourceFields
in class Tap<TConfig,TInput,TOutput>
public Fields getSinkFields()
Tap
getSinkFields
in class Tap<TConfig,TInput,TOutput>
public TupleEntryIterator openForRead(FlowProcess<? extends TConfig> flowProcess, TInput input) throws java.io.IOException
Tap
input
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
input type and instantiate it before calling super.openForRead()
.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.
openForRead
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessinput
- of type Inputjava.io.IOException
- when the resource cannot be openedpublic TupleEntryIterator openForRead(FlowProcess<? extends TConfig> flowProcess) throws java.io.IOException
Tap
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.
openForRead
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessjava.io.IOException
- when the resource cannot be openedpublic TupleEntryCollector openForWrite(FlowProcess<? extends TConfig> flowProcess, TOutput output) throws java.io.IOException
Tap
This method is used internally and does not honor the SinkMode
setting. If SinkMode is
SinkMode.REPLACE
, this call may fail. See Tap.openForWrite(cascading.flow.FlowProcess)
.
output
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
output type and instantiate it before calling super.openForWrite()
.
openForWrite
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessoutput
- of type Outputjava.io.IOException
- when the resource cannot be openedpublic TupleEntryCollector openForWrite(FlowProcess<? extends TConfig> flowProcess) throws java.io.IOException
Tap
This method is for user application use and does honor the SinkMode.REPLACE
settings. That is, if
SinkMode is set to SinkMode.REPLACE
the underlying resource will be deleted.
Note if SinkMode.UPDATE
is set, the resource will not be deleted.
openForWrite
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessjava.io.IOException
- when the resource cannot be openedpublic Scope outgoingScopeFor(java.util.Set<Scope> incomingScopes)
ScopedElement
outgoingScopeFor
in interface ScopedElement
outgoingScopeFor
in class Tap<TConfig,TInput,TOutput>
incomingScopes
- of type Setpublic Fields retrieveSourceFields(FlowProcess<? extends TConfig> flowProcess)
Tap
retrieveSourceFields
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcesspublic void presentSourceFields(FlowProcess<? extends TConfig> flowProcess, Fields fields)
presentSourceFields
in class Tap<TConfig,TInput,TOutput>
public Fields retrieveSinkFields(FlowProcess<? extends TConfig> flowProcess)
Tap
retrieveSinkFields
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcesspublic void presentSinkFields(FlowProcess<? extends TConfig> flowProcess, Fields fields)
presentSinkFields
in class Tap<TConfig,TInput,TOutput>
public Fields resolveIncomingOperationArgumentFields(Scope incomingScope)
ScopedElement
resolveIncomingOperationArgumentFields
in interface ScopedElement
resolveIncomingOperationArgumentFields
in class Tap<TConfig,TInput,TOutput>
incomingScope
- of type Scopepublic Fields resolveIncomingOperationPassThroughFields(Scope incomingScope)
ScopedElement
resolveIncomingOperationPassThroughFields
in interface ScopedElement
resolveIncomingOperationPassThroughFields
in class Tap<TConfig,TInput,TOutput>
incomingScope
- of type Scopepublic java.lang.String getFullIdentifier(FlowProcess<? extends TConfig> flowProcess)
Tap
getFullIdentifier
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcesspublic java.lang.String getFullIdentifier(TConfig conf)
Tap
getFullIdentifier
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configpublic boolean createResource(FlowProcess<? extends TConfig> flowProcess) throws java.io.IOException
Tap
createResource
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessjava.io.IOException
- when there is an error making directoriespublic boolean createResource(TConfig conf) throws java.io.IOException
Tap
createResource
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
- when there is an error making directoriespublic boolean deleteResource(FlowProcess<? extends TConfig> flowProcess) throws java.io.IOException
Tap
deleteResource
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessjava.io.IOException
- when the resource cannot be deletedpublic boolean deleteResource(TConfig conf) throws java.io.IOException
Tap
deleteResource
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
- when the resource cannot be deletedpublic boolean prepareResourceForRead(TConfig conf) throws java.io.IOException
Tap
This method will be called client side so that any remote or external resources can be initialized.
If this method returns false
, an exception will be thrown halting the current Flow.
In most cases, resource initialization should happen in the Tap.openForRead(FlowProcess, Object)
method.
This allows for initialization of cluster side resources, like a JDBC driver used to read data from a database, that cannot be passed client to cluster.
prepareResourceForRead
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
public boolean prepareResourceForWrite(TConfig conf) throws java.io.IOException
Tap
This method will be called once client side so that any remote or external resources can be initialized.
If this method returns false
, an exception will be thrown halting the current Flow.
In most cases, resource initialization should happen in the Tap.openForWrite(FlowProcess, Object)
method.
This allows for initialization of cluster side resources, like a JDBC driver used to write data to a database, that cannot be passed client to cluster.
In the above JDBC example, overriding this method will allow for testing for the existence of and/or creating a remote table used by all individual cluster side tasks.
prepareResourceForWrite
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
public boolean commitResource(TConfig conf) throws java.io.IOException
Tap
See Tap.rollbackResource(Object)
to handle cleanup in the face of failures.
This method is invoked once client side and not in the cluster, if any.
If other sink Tap instance in a given Flow fail on commitResource after called on this instance, rollbackResource will not be called.
commitResource
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
public boolean rollbackResource(TConfig conf) throws java.io.IOException
Tap
See Tap.commitResource(Object)
to handle cleanup when the write has successfully completed.
This method is invoked once client side and not in the cluster, if any.
rollbackResource
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
public boolean resourceExists(FlowProcess<? extends TConfig> flowProcess) throws java.io.IOException
Tap
resourceExists
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessjava.io.IOException
- when the status cannot be determinedpublic boolean resourceExists(TConfig conf) throws java.io.IOException
Tap
resourceExists
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
- when the status cannot be determinedpublic long getModifiedTime(FlowProcess<? extends TConfig> flowProcess) throws java.io.IOException
Tap
If the resource does not exist, returns zero (0).
If the resource is continuous, returns Long.MAX_VALUE
.
getModifiedTime
in class Tap<TConfig,TInput,TOutput>
flowProcess
- of type FlowProcessjava.io.IOException
public long getModifiedTime(TConfig conf) throws java.io.IOException
Tap
If the resource does not exist, returns zero (0).
If the resource is continuous, returns Long.MAX_VALUE
.
getModifiedTime
in class Tap<TConfig,TInput,TOutput>
conf
- of type Configjava.io.IOException
public SinkMode getSinkMode()
Tap
SinkMode
}of this Tap object.getSinkMode
in class Tap<TConfig,TInput,TOutput>
public boolean isKeep()
Tap
public boolean isReplace()
Tap
public boolean isUpdate()
Tap
Tap.createResource(Object)
, when the Flow is started.public boolean isSink()
Tap
public boolean isSource()
Tap
public boolean isTemporary()
Tap
isTemporary
in class Tap<TConfig,TInput,TOutput>
public ConfigDef getConfigDef()
Tap
ConfigDef
instance that allows for local properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the configDef will not show up in any Flow
or FlowStep
process
level configuration, but will override any of those values as seen by the current Tap instance method call where a
FlowProcess is provided except for the Tap.sourceConfInit(cascading.flow.FlowProcess, Object)
and
Tap.sinkConfInit(cascading.flow.FlowProcess, Object)
methods.
That is, the *confInit
methods are called before any ConfigDef is applied, so any values placed into
a ConfigDef instance will not be visible to them.
getConfigDef
in interface ScopedElement
getConfigDef
in class Tap<TConfig,TInput,TOutput>
public boolean hasConfigDef()
Tap
true
if there are properties in the configDef instance.hasConfigDef
in interface ScopedElement
hasConfigDef
in class Tap<TConfig,TInput,TOutput>
public ConfigDef getNodeConfigDef()
Tap
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the nodeConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowNode
(in Apache Tez the Vertex configuration). Any value set in the
nodeConfigDef will be overridden by the pipe local #getConfigDef
instance.
Use this method to tweak properties in the process node this tap instance is planned into.
getNodeConfigDef
in interface ScopedElement
getNodeConfigDef
in class Tap<TConfig,TInput,TOutput>
public boolean hasNodeConfigDef()
Tap
true
if there are properties in the nodeConfigDef instance.hasNodeConfigDef
in interface ScopedElement
hasNodeConfigDef
in class Tap<TConfig,TInput,TOutput>
public ConfigDef getStepConfigDef()
Tap
ConfigDef
instance that allows for process level properties to be set and made available via
a resulting FlowProcess
instance when the tap is invoked.
Any properties set on the stepConfigDef will not show up in any Flow configuration, but will show up in
the current process FlowStep
(in Hadoop the MapReduce jobconf). Any value set in the
stepConfigDef will be overridden by the tap local #getConfigDef
instance.
Use this method to tweak properties in the process step this tap instance is planned into.
Note the *confInit
methods are called before any ConfigDef is applied, so any values placed into
a ConfigDef instance will not be visible to them.
getStepConfigDef
in interface ScopedElement
getStepConfigDef
in class Tap<TConfig,TInput,TOutput>
public boolean hasStepConfigDef()
Tap
true
if there are properties in the stepConfigDef instance.hasStepConfigDef
in interface ScopedElement
hasStepConfigDef
in class Tap<TConfig,TInput,TOutput>
public boolean equals(java.lang.Object object)
Copyright © 2007-2017 Cascading Maintainers. All Rights Reserved.