cascading.tap

Class MultiSourceTap<Child extends Tap,Config,Input>

java.lang.Object

cascading.tap.Tap<Config,Input,java.lang.Void>

cascading.tap.SourceTap<Config,Input>

cascading.tap.MultiSourceTap<Child,Config,Input>

All Implemented Interfaces:

FlowElement, ScopedElement, CompositeTap<Child>, Traceable, java.io.Serializable

public class MultiSourceTap<Child extends Tap,Config,Input>
extends SourceTap<Config,Input>
implements CompositeTap<Child>

Class MultiSourceTap is used to tie multiple Tap instances into a single resource. Effectively this will allow multiple files to be concatenated into the requesting pipe assembly, if they all share the same Scheme instance.

Note that order is not maintained by virtue of the underlying model. If order is necessary, use a unique sequence key to span the resources, like a line number.

Note that if multiple input files have the same Scheme (like TextLine), they may not contain the same semi-structure internally. For example, one file might be an Apache log file, and another might be a Log4J log file. If each one should be parsed differently, then they must be handled by different pipe assembly branches.

As of 3.3, MultiSourceTap can aggregate PartitionTap and PartitionTap instances.

But it may not be safe to aggregate GlobHfs and PartitionTap instances as GlobHfs identifiers cannot be fully resolved, preventing the cluster side runtime from distinguishing which Tap instance to open for reading.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	commonPrefix
protected Trie<Child>	prefixMap
protected Child[]	taps

Constructor Summary

Constructors

Modifier	Constructor and Description
	MultiSourceTap(Child... taps) Constructor MultiSourceTap creates a new MultiSourceTap instance.
protected	MultiSourceTap(Scheme<Config,Input,?,?,?> scheme)

Method Summary

Modifier and Type	Method and Description
boolean	equals(java.lang.Object object)
protected Child	findMatchingTap(FlowProcess<? extends Config> flowProcess)
java.util.Iterator<Child>	getChildTaps()
java.lang.String	getIdentifier() Method getIdentifier returns a String representing the resource this Tap instance represents.
long	getModifiedTime(Config conf) Returns the most current modified time.
long	getNumChildTaps()
Scheme	getScheme() Method getScheme returns the scheme of this Tap object.
protected Trie<Child>	getTapPrefixMap(FlowProcess<? extends Config> flowProcess)
protected Child[]	getTaps() Method getTaps returns the taps of this MultiTap object.
protected java.lang.String	getTapsCommonPrefix(FlowProcess<? extends Config> flowProcess)
int	hashCode()
boolean	isReplace() Method isReplace indicates whether the resource represented by this instance should be deleted if it already exists when the Flow is started.
TupleEntryIterator	openForRead(FlowProcess<? extends Config> flowProcess, Input input) Method openForRead opens the resource represented by this Tap instance for reading.
boolean	resourceExists(Config conf) Method resourceExists returns true if the path represented by this instance exists.
void	sourceConfInit(FlowProcess<? extends Config> process, Config conf) Method sourceConfInit initializes this instance as a source.
java.lang.String	toString()

Methods inherited from class cascading.tap.SourceTap

commitResource, createResource, deleteResource, getSinkFields, isSink, openForWrite, prepareResourceForWrite, rollbackResource, sinkConfInit

Methods inherited from class cascading.tap.Tap

createResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, prepareResourceForRead, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

taps

protected Child extends Tap[] taps

prefixMap

protected transient Trie<Child extends Tap> prefixMap

commonPrefix

protected transient java.lang.String commonPrefix

Constructor Detail

MultiSourceTap

protected MultiSourceTap(Scheme<Config,Input,?,?,?> scheme)

MultiSourceTap

@ConstructorProperties(value="taps")
public MultiSourceTap(Child... taps)

Constructor MultiSourceTap creates a new MultiSourceTap instance.

Parameters:

taps - of type Tap...

Method Detail

getTaps

protected Child[] getTaps()

Method getTaps returns the taps of this MultiTap object.

Returns:

the taps (type Tap[]) of this MultiTap object.

getChildTaps

public java.util.Iterator<Child> getChildTaps()

Specified by:

getChildTaps in interface CompositeTap<Child extends Tap>

getNumChildTaps

public long getNumChildTaps()

Specified by:

getNumChildTaps in interface CompositeTap<Child extends Tap>

getIdentifier

public java.lang.String getIdentifier()

Description copied from class: Tap

Method getIdentifier returns a String representing the resource this Tap instance represents.

Often, if the tap accesses a filesystem, the identifier is nothing more than the path to the file or directory. In other cases it may be a an URL or URI representing a connection string or remote resource.

Any two Tap instances having the same value for the identifier are considered equal.

Specified by:

getIdentifier in class Tap<Config,Input,java.lang.Void>

Returns:

String

getScheme

public Scheme getScheme()

Description copied from class: Tap

Method getScheme returns the scheme of this Tap object.

Overrides:

getScheme in class Tap<Config,Input,java.lang.Void>

Returns:

the scheme (type Scheme) of this Tap object.

isReplace

public boolean isReplace()

Description copied from class: Tap

Method isReplace indicates whether the resource represented by this instance should be deleted if it already exists when the Flow is started.

Overrides:

isReplace in class Tap<Config,Input,java.lang.Void>

Returns:

boolean

sourceConfInit

public void sourceConfInit(FlowProcess<? extends Config> process,
                           Config conf)

Description copied from class: Tap

Method sourceConfInit initializes this instance as a source.

This method maybe called more than once if this Tap instance is used outside the scope of a Flow instance or if it participates in multiple times in a given Flow or across different Flows in a Cascade.

In the context of a Flow, it will be called after FlowListener.onStarting(cascading.flow.Flow)

Note that no resources or services should be modified by this method.

Overrides:

sourceConfInit in class Tap<Config,Input,java.lang.Void>

Parameters:

process - of type FlowProcess

conf - of type Config

resourceExists

public boolean resourceExists(Config conf)
                       throws java.io.IOException

Description copied from class: Tap

Method resourceExists returns true if the path represented by this instance exists.

Specified by:

resourceExists in class Tap<Config,Input,java.lang.Void>

Parameters:

conf - of type Config

Returns:

true if the underlying resource already exists

Throws:

java.io.IOException - when the status cannot be determined

getModifiedTime

public long getModifiedTime(Config conf)
                     throws java.io.IOException

Returns the most current modified time.

Specified by:

getModifiedTime in class Tap<Config,Input,java.lang.Void>

Parameters:

conf - of type Config

Returns:

The date this resource was last modified.

Throws:

java.io.IOException

openForRead

public TupleEntryIterator openForRead(FlowProcess<? extends Config> flowProcess,
                                      Input input)
                               throws java.io.IOException

Description copied from class: Tap

Method openForRead opens the resource represented by this Tap instance for reading.

input value may be null, if so, sub-classes must inquire with the underlying Scheme via Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object) to get the proper input type and instantiate it before calling super.openForRead().

Note the returned iterator will return the same instance of TupleEntry on every call, thus a copy must be made of either the TupleEntry or the underlying Tuple instance if they are to be stored in a Collection.

Specified by:

openForRead in class Tap<Config,Input,java.lang.Void>

Parameters:

flowProcess - of type FlowProcess

input - of type Input

Returns:

TupleEntryIterator

Throws:

java.io.IOException - when the resource cannot be opened

findMatchingTap

protected Child findMatchingTap(FlowProcess<? extends Config> flowProcess)

getTapsCommonPrefix

protected java.lang.String getTapsCommonPrefix(FlowProcess<? extends Config> flowProcess)

getTapPrefixMap

protected Trie<Child> getTapPrefixMap(FlowProcess<? extends Config> flowProcess)

equals

public boolean equals(java.lang.Object object)

Overrides:

equals in class Tap<Config,Input,java.lang.Void>

hashCode

public int hashCode()

Overrides:

hashCode in class Tap<Config,Input,java.lang.Void>

toString

public java.lang.String toString()

Overrides:

toString in class Tap<Config,Input,java.lang.Void>