cascading.tap

Class SourceTap<Config,Input>

java.lang.Object

cascading.tap.Tap<Config,Input,java.lang.Void>

cascading.tap.SourceTap<Config,Input>

All Implemented Interfaces:

FlowElement, ScopedElement, Traceable, java.io.Serializable

Direct Known Subclasses:

MultiSourceTap

public abstract class SourceTap<Config,Input>
extends Tap<Config,Input,java.lang.Void>

Class SourceTap is an optional base class for source only Taps.

Some Tap instances may only be sources (as opposed to being a sink). These types should subclass SourceTap for convenience or set isSink() to false in a custom Tap sub-class.

See Also:

Serialized Form

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	SourceTap()
protected	SourceTap(Scheme<Config,Input,?,?,?> scheme)

Method Summary

Modifier and Type	Method and Description
boolean	commitResource(Config conf) Method commitResource allows the underlying resource to be notified when all write processing is successful so that any additional cleanup or processing may be completed.
boolean	createResource(Config conf) Method createResource creates the underlying resource.
boolean	deleteResource(Config conf) Method deleteResource deletes the resource represented by this instance.
Fields	getSinkFields() Method getSinkFields returns the sinkFields of this Tap object.
boolean	isSink() Method isSink returns true if this Tap instance can be used as a sink.
TupleEntryCollector	openForWrite(FlowProcess<? extends Config> flowProcess, java.lang.Void output) Method openForWrite opens the resource represented by this Tap instance for writing.
boolean	prepareResourceForWrite(Config conf) Method prepareResourceForWrite allows the underlying resource to be notified when writing will begin.
boolean	rollbackResource(Config conf) Method rollbackResource allows the underlying resource to be notified when any write processing has failed or was stopped so that any cleanup may be started.
void	sinkConfInit(FlowProcess<? extends Config> flowProcess, Config conf) Method sinkConfInit initializes this instance as a sink.

Methods inherited from class cascading.tap.Tap

createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, equals, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getIdentifier, getModifiedTime, getModifiedTime, getNodeConfigDef, getScheme, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSource, isTemporary, isUpdate, openForRead, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, prepareResourceForRead, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, sourceConfInit, spliterator, splititerator, taps, toString, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

SourceTap

protected SourceTap()

SourceTap

protected SourceTap(Scheme<Config,Input,?,?,?> scheme)

Method Detail

getSinkFields

public Fields getSinkFields()

Description copied from class: Tap

Method getSinkFields returns the sinkFields of this Tap object.

Overrides:

getSinkFields in class Tap<Config,Input,java.lang.Void>

Returns:

the sinkFields (type Fields) of this Tap object.

isSink

public final boolean isSink()

Description copied from class: Tap

Method isSink returns true if this Tap instance can be used as a sink.

Overrides:

isSink in class Tap<Config,Input,java.lang.Void>

Returns:

boolean

deleteResource

public boolean deleteResource(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method deleteResource deletes the resource represented by this instance.

Specified by:

deleteResource in class Tap<Config,Input,java.lang.Void>

Parameters:

conf - of type Config

Returns:

boolean

Throws:

java.io.IOException - when the resource cannot be deleted

sinkConfInit

public void sinkConfInit(FlowProcess<? extends Config> flowProcess,
                         Config conf)

Description copied from class: Tap

Method sinkConfInit initializes this instance as a sink.

This method maybe called more than once if this Tap instance is used outside the scope of a Flow instance or if it participates in multiple times in a given Flow or across different Flows in a Cascade.

Note this method will be called in context of this Tap being used as a traditional 'sink' and as a 'trap'.

In the context of a Flow, it will be called after FlowListener.onStarting(cascading.flow.Flow)

Note that no resources or services should be modified by this method. If this Tap instance returns true for Tap.isReplace(), then Tap.deleteResource(Object) will be called by the parent Flow.

Overrides:

sinkConfInit in class Tap<Config,Input,java.lang.Void>

Parameters:

flowProcess - of type FlowProcess

conf - of type Config

prepareResourceForWrite

public boolean prepareResourceForWrite(Config conf)
                                throws java.io.IOException

Description copied from class: Tap

Method prepareResourceForWrite allows the underlying resource to be notified when writing will begin.

This method will be called once client side so that any remote or external resources can be initialized.

If this method returns false, an exception will be thrown halting the current Flow.

In most cases, resource initialization should happen in the Tap.openForWrite(FlowProcess, Object) method.

This allows for initialization of cluster side resources, like a JDBC driver used to write data to a database, that cannot be passed client to cluster.

In the above JDBC example, overriding this method will allow for testing for the existence of and/or creating a remote table used by all individual cluster side tasks.

Overrides:

prepareResourceForWrite in class Tap<Config,Input,java.lang.Void>

Parameters:

conf - of type Config

Returns:

returns true if successful

Throws:

java.io.IOException

createResource

public boolean createResource(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method createResource creates the underlying resource.

Specified by:

createResource in class Tap<Config,Input,java.lang.Void>

Parameters:

conf - of type Config

Returns:

boolean

Throws:

java.io.IOException - when there is an error making directories

commitResource

public boolean commitResource(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method commitResource allows the underlying resource to be notified when all write processing is successful so that any additional cleanup or processing may be completed.

See Tap.rollbackResource(Object) to handle cleanup in the face of failures.

This method is invoked once client side and not in the cluster, if any.

If other sink Tap instance in a given Flow fail on commitResource after called on this instance, rollbackResource will not be called.

Overrides:

commitResource in class Tap<Config,Input,java.lang.Void>

Parameters:

conf - of type Config

Returns:

returns true if successful

Throws:

java.io.IOException

rollbackResource

public boolean rollbackResource(Config conf)
                         throws java.io.IOException

Description copied from class: Tap

Method rollbackResource allows the underlying resource to be notified when any write processing has failed or was stopped so that any cleanup may be started.

See Tap.commitResource(Object) to handle cleanup when the write has successfully completed.

This method is invoked once client side and not in the cluster, if any.

Overrides:

rollbackResource in class Tap<Config,Input,java.lang.Void>

Parameters:

conf - of type Config

Returns:

returns true if successful

Throws:

java.io.IOException

openForWrite

public TupleEntryCollector openForWrite(FlowProcess<? extends Config> flowProcess,
                                        java.lang.Void output)
                                 throws java.io.IOException

Description copied from class: Tap

Method openForWrite opens the resource represented by this Tap instance for writing.

This method is used internally and does not honor the SinkMode setting. If SinkMode is SinkMode.REPLACE, this call may fail. See Tap.openForWrite(cascading.flow.FlowProcess).

output value may be null, if so, sub-classes must inquire with the underlying Scheme via Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object) to get the proper output type and instantiate it before calling super.openForWrite().

Specified by:

openForWrite in class Tap<Config,Input,java.lang.Void>

Parameters:

flowProcess - of type FlowProcess

output - of type Output

Returns:

TupleEntryCollector

Throws:

java.io.IOException - when the resource cannot be opened