public abstract class BasePartitionTap<Config,Input,Output> extends Tap<Config,Input,Output>
Modifier and Type | Class and Description |
---|---|
static class |
BasePartitionTap.Counters |
class |
BasePartitionTap.PartitionCollector |
static class |
BasePartitionTap.PartitionScheme<Config,Input,Output> |
Modifier and Type | Field and Description |
---|---|
protected boolean |
keepParentOnDelete
Field keepParentOnDelete
|
protected static int |
OPEN_WRITES_THRESHOLD_DEFAULT
Field OPEN_FILES_THRESHOLD_DEFAULT
|
protected int |
openWritesThreshold
Field openTapsThreshold
|
protected Tap |
parent
Field parent
|
protected Partition |
partition
Field partition
|
protected List<cascading.tap.partition.PartitionTapFilter> |
sourcePartitionFilters
Field sourcePartitionFilters
|
Modifier | Constructor and Description |
---|---|
protected |
BasePartitionTap(Tap parent,
Partition partition,
int openWritesThreshold) |
protected |
BasePartitionTap(Tap parent,
Partition partition,
SinkMode sinkMode) |
protected |
BasePartitionTap(Tap parent,
Partition partition,
SinkMode sinkMode,
boolean keepParentOnDelete,
int openWritesThreshold) |
Modifier and Type | Method and Description |
---|---|
void |
addSourcePartitionFilter(Fields argumentSelector,
Filter filter)
Add a
Filter with its associated argument selector when using this PartitionTap as a source. |
boolean |
commitResource(Config conf)
Method commitResource allows the underlying resource to be notified when all write processing is
successful so that any additional cleanup or processing may be completed.
|
boolean |
createResource(Config conf)
Method createResource creates the underlying resource.
|
protected abstract TupleEntrySchemeCollector |
createTupleEntrySchemeCollector(FlowProcess<? extends Config> flowProcess,
Tap parent,
String path,
long sequence) |
protected abstract TupleEntrySchemeIterator |
createTupleEntrySchemeIterator(FlowProcess<? extends Config> flowProcess,
Tap parent,
String path,
Input input) |
boolean |
deleteResource(Config conf)
Method deleteResource deletes the resource represented by this instance.
|
boolean |
equals(Object object) |
String[] |
getChildPartitionIdentifiers(FlowProcess<? extends Config> flowProcess,
boolean fullyQualified)
Method getChildPartitionIdentifiers returns an array of all identifiers for all available partitions.
|
protected abstract String |
getCurrentIdentifier(FlowProcess<? extends Config> flowProcess) |
protected String[] |
getFilteredPartitionIdentifiers(FlowProcess<? extends Config> flowProcess,
String[] childIdentifiers) |
String |
getIdentifier()
Method getIdentifier returns a String representing the resource this Tap instance represents.
|
long |
getModifiedTime(Config conf)
Method getModifiedTime returns the date this resource was last modified.
|
int |
getOpenWritesThreshold()
Method getOpenWritesThreshold returns the openTapsThreshold of this PartitionTap object.
|
Tap |
getParent()
Method getParent returns the parent Tap of this PartitionTap object.
|
Partition |
getPartition()
Method getPartition returns the
Partition instance used by this PartitionTap |
int |
hashCode() |
TupleEntryIterator |
openForRead(FlowProcess<? extends Config> flowProcess,
Input input)
Method openForRead opens the resource represented by this Tap instance for reading.
|
TupleEntryCollector |
openForWrite(FlowProcess<? extends Config> flowProcess,
Output output)
Method openForWrite opens the resource represented by this Tap instance for writing.
|
boolean |
prepareResourceForRead(Config conf)
Method prepareResourceForRead allows the underlying resource to be notified when reading will begin.
|
boolean |
prepareResourceForWrite(Config conf)
Method prepareResourceForWrite allows the underlying resource to be notified when writing will begin.
|
boolean |
resourceExists(Config conf)
Method resourceExists returns true if the path represented by this instance exists.
|
boolean |
rollbackResource(Config conf)
Method rollbackResource allows the underlying resource to be notified when any write processing has failed or
was stopped so that any cleanup may be started.
|
String |
toString() |
createResource, deleteResource, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForWrite, outgoingScopeFor, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, sinkConfInit, sourceConfInit, taps
protected static final int OPEN_WRITES_THRESHOLD_DEFAULT
protected final List<cascading.tap.partition.PartitionTapFilter> sourcePartitionFilters
protected boolean keepParentOnDelete
protected int openWritesThreshold
protected BasePartitionTap(Tap parent, Partition partition, int openWritesThreshold)
protected BasePartitionTap(Tap parent, Partition partition, SinkMode sinkMode)
protected BasePartitionTap(Tap parent, Partition partition, SinkMode sinkMode, boolean keepParentOnDelete, int openWritesThreshold)
protected abstract TupleEntrySchemeCollector createTupleEntrySchemeCollector(FlowProcess<? extends Config> flowProcess, Tap parent, String path, long sequence) throws IOException
IOException
protected abstract TupleEntrySchemeIterator createTupleEntrySchemeIterator(FlowProcess<? extends Config> flowProcess, Tap parent, String path, Input input) throws IOException
IOException
public Tap getParent()
public Partition getPartition()
Partition
instance used by this PartitionTappublic String[] getChildPartitionIdentifiers(FlowProcess<? extends Config> flowProcess, boolean fullyQualified) throws IOException
flowProcess
- of type FlowProcessfullyQualified
- of type booleanIOException
protected String[] getFilteredPartitionIdentifiers(FlowProcess<? extends Config> flowProcess, String[] childIdentifiers)
public void addSourcePartitionFilter(Fields argumentSelector, Filter filter)
Filter
with its associated argument selector when using this PartitionTap as a source. On read, each
child identifier is converted to a Tuple
using the provided Partition
. Each Filter
will be
applied to the Tuple
so that the input paths can be filtered to only accept those required for the
Flow
.argumentSelector
- field selector that selects Filter arguments from the input Tuplefilter
- Filter to be applied to each input Tuplepublic String getIdentifier()
Tap
getIdentifier
in class Tap<Config,Input,Output>
protected abstract String getCurrentIdentifier(FlowProcess<? extends Config> flowProcess)
public int getOpenWritesThreshold()
public TupleEntryCollector openForWrite(FlowProcess<? extends Config> flowProcess, Output output) throws IOException
Tap
SinkMode
setting. If SinkMode is
SinkMode.REPLACE
, this call may fail. See Tap.openForWrite(cascading.flow.FlowProcess)
.
output
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sinkConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
output type and instantiate it before calling super.openForWrite()
.openForWrite
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessoutput
- of type OutputIOException
- when the resource cannot be openedpublic TupleEntryIterator openForRead(FlowProcess<? extends Config> flowProcess, Input input) throws IOException
Tap
input
value may be null, if so, sub-classes must inquire with the underlying Scheme
via Scheme.sourceConfInit(cascading.flow.FlowProcess, Tap, Object)
to get the proper
input type and instantiate it before calling super.openForRead()
.
Note the returned iterator will return the same instance of TupleEntry
on every call,
thus a copy must be made of either the TupleEntry or the underlying Tuple
instance if they are to be
stored in a Collection.openForRead
in class Tap<Config,Input,Output>
flowProcess
- of type FlowProcessinput
- of type InputIOException
- when the resource cannot be openedpublic boolean createResource(Config conf) throws IOException
Tap
createResource
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
- when there is an error making directoriespublic boolean deleteResource(Config conf) throws IOException
Tap
deleteResource
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
- when the resource cannot be deletedpublic boolean prepareResourceForRead(Config conf) throws IOException
Tap
false
, an exception will be thrown halting the current Flow.
In most cases, resource initialization should happen in the Tap.openForRead(FlowProcess, Object)
method.
This allows for initialization of cluster side resources, like a JDBC driver used to read data from a database,
that cannot be passed client to cluster.prepareResourceForRead
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
public boolean prepareResourceForWrite(Config conf) throws IOException
Tap
false
, an exception will be thrown halting the current Flow.
In most cases, resource initialization should happen in the Tap.openForWrite(FlowProcess, Object)
method.
This allows for initialization of cluster side resources, like a JDBC driver used to write data to a database,
that cannot be passed client to cluster.
In the above JDBC example, overriding this method will allow for testing for the existence of and/or creating
a remote table used by all individual cluster side tasks.prepareResourceForWrite
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
public boolean commitResource(Config conf) throws IOException
Tap
Tap.rollbackResource(Object)
to handle cleanup in the face of failures.
This method is invoked once client side and not in the cluster, if any.
If other sink Tap instance in a given Flow fail on commitResource after called on this instance,
rollbackResource will not be called.commitResource
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
public boolean rollbackResource(Config conf) throws IOException
Tap
Tap.commitResource(Object)
to handle cleanup when the write has successfully completed.
This method is invoked once client side and not in the cluster, if any.rollbackResource
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
public boolean resourceExists(Config conf) throws IOException
Tap
resourceExists
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
- when the status cannot be determinedpublic long getModifiedTime(Config conf) throws IOException
Tap
getModifiedTime
in class Tap<Config,Input,Output>
conf
- of type ConfigIOException
Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.