cascading.tap.partition

Class BasePartitionTap<Config,Input,Output>

java.lang.Object

cascading.tap.Tap<Config,Input,Output>

cascading.tap.partition.BasePartitionTap<Config,Input,Output>

All Implemented Interfaces:

FlowElement, ScopedElement, FileType<Config>, Traceable, java.io.Serializable

public abstract class BasePartitionTap<Config,Input,Output>
extends Tap<Config,Input,Output>
implements FileType<Config>

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	BasePartitionTap.Counters
class	BasePartitionTap.PartitionCollector
static class	BasePartitionTap.PartitionScheme<Config,Input,Output>

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	keepParentOnDelete Field keepParentOnDelete
protected static int	OPEN_WRITES_THRESHOLD_DEFAULT Field OPEN_FILES_THRESHOLD_DEFAULT
protected int	openWritesThreshold Field openTapsThreshold
protected Tap	parent Field parent
protected Partition	partition Field partition
protected java.util.List<cascading.tap.partition.PartitionTapFilter>	sourcePartitionFilters Field sourcePartitionFilters

Fields inherited from interface cascading.tap.type.FileType

CASCADING_SOURCE_PATH

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BasePartitionTap(Tap parent, Partition partition, int openWritesThreshold)
protected	BasePartitionTap(Tap parent, Partition partition, SinkMode sinkMode)
protected	BasePartitionTap(Tap parent, Partition partition, SinkMode sinkMode, boolean keepParentOnDelete, int openWritesThreshold)

Method Summary

Modifier and Type	Method and Description
void	addSourcePartitionFilter(Fields argumentSelector, Filter filter) Add a Filter with its associated argument selector when using this PartitionTap as a source.
protected FileType<Config>	castFileType()
boolean	commitResource(Config conf) Method commitResource allows the underlying resource to be notified when all write processing is successful so that any additional cleanup or processing may be completed.
boolean	createResource(Config conf) Method createResource creates the underlying resource.
protected abstract TupleEntrySchemeCollector	createTupleEntrySchemeCollector(FlowProcess<? extends Config> flowProcess, Tap parent, java.lang.String path, long sequence)
protected abstract TupleEntrySchemeIterator	createTupleEntrySchemeIterator(FlowProcess<? extends Config> flowProcess, Tap parent, java.lang.String path, Input input)
boolean	deleteResource(Config conf) Method deleteResource deletes the resource represented by this instance.
boolean	equals(java.lang.Object object)
java.lang.String[]	getChildIdentifiers(Config conf) Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.
java.lang.String[]	getChildIdentifiers(Config conf, int depth, boolean fullyQualified) Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.
java.lang.String[]	getChildIdentifiers(FlowProcess<? extends Config> flowProcess) Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.
java.lang.String[]	getChildIdentifiers(FlowProcess<? extends Config> flowProcess, int depth, boolean fullyQualified) Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.
java.lang.String[]	getChildPartitionIdentifiers(FlowProcess<? extends Config> flowProcess, boolean fullyQualified) Method getChildPartitionIdentifiers returns an array of all identifiers for all available partitions.
protected abstract java.lang.String	getCurrentIdentifier(FlowProcess<? extends Config> flowProcess)
protected java.lang.String[]	getFilteredPartitionIdentifiers(FlowProcess<? extends Config> flowProcess, java.lang.String[] childIdentifiers)
java.lang.String	getIdentifier() Method getIdentifier returns a String representing the resource this Tap instance represents.
long	getModifiedTime(Config conf) Method getModifiedTime returns the date this resource was last modified.
int	getOpenWritesThreshold() Method getOpenWritesThreshold returns the openTapsThreshold of this PartitionTap object.
Tap	getParent() Method getParent returns the parent Tap of this PartitionTap object.
Partition	getPartition() Method getPartition returns the Partition instance used by this PartitionTap
long	getSize(Config conf) Method getSize returns the size of the file referenced by this tap.
long	getSize(FlowProcess<? extends Config> flowProcess) Method getSize returns the size of the file referenced by this tap.
int	hashCode()
boolean	isDirectory(Config conf) Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.
boolean	isDirectory(FlowProcess<? extends Config> flowProcess) Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.
TupleEntryIterator	openForRead(FlowProcess<? extends Config> flowProcess, Input input) Method openForRead opens the resource represented by this Tap instance for reading.
TupleEntryCollector	openForWrite(FlowProcess<? extends Config> flowProcess, Output output) Method openForWrite opens the resource represented by this Tap instance for writing.
boolean	prepareResourceForRead(Config conf) Method prepareResourceForRead allows the underlying resource to be notified when reading will begin.
boolean	prepareResourceForWrite(Config conf) Method prepareResourceForWrite allows the underlying resource to be notified when writing will begin.
boolean	resourceExists(Config conf) Method resourceExists returns true if the path represented by this instance exists.
boolean	rollbackResource(Config conf) Method rollbackResource allows the underlying resource to be notified when any write processing has failed or was stopped so that any cleanup may be started.
java.lang.String	toString()

Methods inherited from class cascading.tap.Tap

createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, sinkConfInit, sourceConfInit, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

OPEN_WRITES_THRESHOLD_DEFAULT

protected static final int OPEN_WRITES_THRESHOLD_DEFAULT

Field OPEN_FILES_THRESHOLD_DEFAULT

See Also:

Constant Field Values

parent

protected Tap parent

Field parent

partition

protected Partition partition

Field partition

sourcePartitionFilters

protected final java.util.List<cascading.tap.partition.PartitionTapFilter> sourcePartitionFilters

Field sourcePartitionFilters

keepParentOnDelete

protected boolean keepParentOnDelete

Field keepParentOnDelete

openWritesThreshold

protected int openWritesThreshold

Field openTapsThreshold

Constructor Detail

BasePartitionTap

protected BasePartitionTap(Tap parent,
                           Partition partition,
                           int openWritesThreshold)

BasePartitionTap

protected BasePartitionTap(Tap parent,
                           Partition partition,
                           SinkMode sinkMode)

BasePartitionTap

protected BasePartitionTap(Tap parent,
                           Partition partition,
                           SinkMode sinkMode,
                           boolean keepParentOnDelete,
                           int openWritesThreshold)

Method Detail

createTupleEntrySchemeCollector

protected abstract TupleEntrySchemeCollector createTupleEntrySchemeCollector(FlowProcess<? extends Config> flowProcess,
                                                                             Tap parent,
                                                                             java.lang.String path,
                                                                             long sequence)
                                                                      throws java.io.IOException

Throws:

java.io.IOException

createTupleEntrySchemeIterator

protected abstract TupleEntrySchemeIterator createTupleEntrySchemeIterator(FlowProcess<? extends Config> flowProcess,
                                                                           Tap parent,
                                                                           java.lang.String path,
                                                                           Input input)
                                                                    throws java.io.IOException

Throws:

java.io.IOException

getParent

public Tap getParent()

Method getParent returns the parent Tap of this PartitionTap object.

Returns:

the parent (type Tap) of this PartitionTap object.

getPartition

public Partition getPartition()

Method getPartition returns the Partition instance used by this PartitionTap

Returns:

the partition instance

getChildPartitionIdentifiers

public java.lang.String[] getChildPartitionIdentifiers(FlowProcess<? extends Config> flowProcess,
                                                       boolean fullyQualified)
                                                throws java.io.IOException

Method getChildPartitionIdentifiers returns an array of all identifiers for all available partitions.

This method is used internally to set all incoming paths, override to limit applicable partitions.

Note the returns array may be large.

Parameters:

flowProcess - of type FlowProcess

fullyQualified - of type boolean

Returns:

a String[] of partition identifiers

Throws:

java.io.IOException

getFilteredPartitionIdentifiers

protected java.lang.String[] getFilteredPartitionIdentifiers(FlowProcess<? extends Config> flowProcess,
                                                             java.lang.String[] childIdentifiers)

addSourcePartitionFilter

public void addSourcePartitionFilter(Fields argumentSelector,
                                     Filter filter)

Add a Filter with its associated argument selector when using this PartitionTap as a source. On read, each child identifier is converted to a Tuple using the provided Partition. Each Filter will be applied to the Tuple so that the input paths can be filtered to only accept those required for the Flow.

Parameters:

argumentSelector - field selector that selects Filter arguments from the input Tuple

filter - Filter to be applied to each input Tuple

getIdentifier

public java.lang.String getIdentifier()

Description copied from class: Tap

Method getIdentifier returns a String representing the resource this Tap instance represents.

Often, if the tap accesses a filesystem, the identifier is nothing more than the path to the file or directory. In other cases it may be a an URL or URI representing a connection string or remote resource.

Any two Tap instances having the same value for the identifier are considered equal.

Specified by:

getIdentifier in class Tap<Config,Input,Output>

Returns:

String

getCurrentIdentifier

protected abstract java.lang.String getCurrentIdentifier(FlowProcess<? extends Config> flowProcess)

getOpenWritesThreshold

public int getOpenWritesThreshold()

Method getOpenWritesThreshold returns the openTapsThreshold of this PartitionTap object.

Returns:

the openTapsThreshold (type int) of this PartitionTap object.

openForWrite

public TupleEntryCollector openForWrite(FlowProcess<? extends Config> flowProcess,
                                        Output output)
                                 throws java.io.IOException

Description copied from class: Tap

Method openForWrite opens the resource represented by this Tap instance for writing.

This method is used internally and does not honor the SinkMode setting. If SinkMode is SinkMode.REPLACE, this call may fail. See Tap.openForWrite(cascading.flow.FlowProcess).

output value may be null, if so, sub-classes must inquire with the underlying Scheme via Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object) to get the proper output type and instantiate it before calling super.openForWrite().

Specified by:

openForWrite in class Tap<Config,Input,Output>

Parameters:

flowProcess - of type FlowProcess

output - of type Output

Returns:

TupleEntryCollector

Throws:

java.io.IOException - when the resource cannot be opened

openForRead

public TupleEntryIterator openForRead(FlowProcess<? extends Config> flowProcess,
                                      Input input)
                               throws java.io.IOException

Description copied from class: Tap

Method openForRead opens the resource represented by this Tap instance for reading.

input value may be null, if so, sub-classes must inquire with the underlying Scheme via Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object) to get the proper input type and instantiate it before calling super.openForRead().

Note the returned iterator will return the same instance of TupleEntry on every call, thus a copy must be made of either the TupleEntry or the underlying Tuple instance if they are to be stored in a Collection.

Specified by:

openForRead in class Tap<Config,Input,Output>

Parameters:

flowProcess - of type FlowProcess

input - of type Input

Returns:

TupleEntryIterator

Throws:

java.io.IOException - when the resource cannot be opened

createResource

public boolean createResource(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method createResource creates the underlying resource.

Specified by:

createResource in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

boolean

Throws:

java.io.IOException - when there is an error making directories

deleteResource

public boolean deleteResource(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method deleteResource deletes the resource represented by this instance.

Specified by:

deleteResource in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

boolean

Throws:

java.io.IOException - when the resource cannot be deleted

prepareResourceForRead

public boolean prepareResourceForRead(Config conf)
                               throws java.io.IOException

Description copied from class: Tap

Method prepareResourceForRead allows the underlying resource to be notified when reading will begin.

This method will be called client side so that any remote or external resources can be initialized.

If this method returns false, an exception will be thrown halting the current Flow.

In most cases, resource initialization should happen in the Tap.openForRead(FlowProcess, Object) method.

This allows for initialization of cluster side resources, like a JDBC driver used to read data from a database, that cannot be passed client to cluster.

Overrides:

prepareResourceForRead in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

returns true if successful

Throws:

java.io.IOException

prepareResourceForWrite

public boolean prepareResourceForWrite(Config conf)
                                throws java.io.IOException

Description copied from class: Tap

Method prepareResourceForWrite allows the underlying resource to be notified when writing will begin.

This method will be called once client side so that any remote or external resources can be initialized.

If this method returns false, an exception will be thrown halting the current Flow.

In most cases, resource initialization should happen in the Tap.openForWrite(FlowProcess, Object) method.

This allows for initialization of cluster side resources, like a JDBC driver used to write data to a database, that cannot be passed client to cluster.

In the above JDBC example, overriding this method will allow for testing for the existence of and/or creating a remote table used by all individual cluster side tasks.

Overrides:

prepareResourceForWrite in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

returns true if successful

Throws:

java.io.IOException

commitResource

public boolean commitResource(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method commitResource allows the underlying resource to be notified when all write processing is successful so that any additional cleanup or processing may be completed.

See Tap.rollbackResource(Object) to handle cleanup in the face of failures.

This method is invoked once client side and not in the cluster, if any.

If other sink Tap instance in a given Flow fail on commitResource after called on this instance, rollbackResource will not be called.

Overrides:

commitResource in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

returns true if successful

Throws:

java.io.IOException

rollbackResource

public boolean rollbackResource(Config conf)
                         throws java.io.IOException

Description copied from class: Tap

Method rollbackResource allows the underlying resource to be notified when any write processing has failed or was stopped so that any cleanup may be started.

See Tap.commitResource(Object) to handle cleanup when the write has successfully completed.

This method is invoked once client side and not in the cluster, if any.

Overrides:

rollbackResource in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

returns true if successful

Throws:

java.io.IOException

resourceExists

public boolean resourceExists(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method resourceExists returns true if the path represented by this instance exists.

Specified by:

resourceExists in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

true if the underlying resource already exists

Throws:

java.io.IOException - when the status cannot be determined

getModifiedTime

public long getModifiedTime(Config conf)
                     throws java.io.IOException

Description copied from class: Tap

Method getModifiedTime returns the date this resource was last modified.

If the resource does not exist, returns zero (0).

If the resource is continuous, returns Long.MAX_VALUE.

Specified by:

getModifiedTime in class Tap<Config,Input,Output>

Parameters:

conf - of type Config

Returns:

The date this resource was last modified.

Throws:

java.io.IOException

isDirectory

public boolean isDirectory(FlowProcess<? extends Config> flowProcess)
                    throws java.io.IOException

Description copied from interface: FileType

Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.

Specified by:

isDirectory in interface FileType<Config>

Returns:

boolean

Throws:

java.io.IOException

isDirectory

public boolean isDirectory(Config conf)
                    throws java.io.IOException

Description copied from interface: FileType

Method isDirectory returns true if the underlying resource represents a directory or folder instead of an individual file.

Specified by:

isDirectory in interface FileType<Config>

Parameters:

conf - of Conf

Returns:

boolean

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(FlowProcess<? extends Config> flowProcess)
                                       throws java.io.IOException

Description copied from interface: FileType

Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.

This method will skip Hadoop log directories (_log).

Specified by:

getChildIdentifiers in interface FileType<Config>

Parameters:

flowProcess - of type FlowProcess

Returns:

String[]

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(Config conf)
                                       throws java.io.IOException

Description copied from interface: FileType

Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.

This method will skip Hadoop log directories (_log).

Specified by:

getChildIdentifiers in interface FileType<Config>

Parameters:

conf - of type Conf

Returns:

String[]

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(FlowProcess<? extends Config> flowProcess,
                                              int depth,
                                              boolean fullyQualified)
                                       throws java.io.IOException

Description copied from interface: FileType

Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.

This method will skip Hadoop log directories (_log).

Note that all files encountered up to depth will be returned, and no directories unless the directory is the final path at max depth.

Specified by:

getChildIdentifiers in interface FileType<Config>

Returns:

String{}

Throws:

java.io.IOException

getChildIdentifiers

public java.lang.String[] getChildIdentifiers(Config conf,
                                              int depth,
                                              boolean fullyQualified)
                                       throws java.io.IOException

Description copied from interface: FileType

Method getChildIdentifiers returns an array of child identifiers if this resource is a directory.

This method will skip Hadoop log directories (_log).

Note that all files encountered up to depth will be returned, and no directories unless the directory is the final path at max depth.

Specified by:

getChildIdentifiers in interface FileType<Config>

Parameters:

conf - of type Conf

depth - the max depth to return.

fullyQualified - true if the returned paths should be fully qualified

Returns:

String[]

Throws:

java.io.IOException

getSize

public long getSize(FlowProcess<? extends Config> flowProcess)
             throws java.io.IOException

Description copied from interface: FileType

Method getSize returns the size of the file referenced by this tap.

Specified by:

getSize in interface FileType<Config>

Returns:

The size of the file reference by this tap.

Throws:

java.io.IOException

getSize

public long getSize(Config conf)
             throws java.io.IOException

Description copied from interface: FileType

Method getSize returns the size of the file referenced by this tap.

Specified by:

getSize in interface FileType<Config>

Parameters:

conf - of type Config

Returns:

The size of the file reference by this tap.

Throws:

java.io.IOException

castFileType

protected FileType<Config> castFileType()

equals

public boolean equals(java.lang.Object object)

Overrides:

equals in class Tap<Config,Input,Output>

hashCode

public int hashCode()

Overrides:

hashCode in class Tap<Config,Input,Output>

toString

public java.lang.String toString()

Overrides:

toString in class Tap<Config,Input,Output>