public class PartitionTap extends BasePartitionTap<Configuration,RecordReader,OutputCollector>
Tuple instance.
The constructor takes a Hfs Tap and a Partition
implementation. This allows Tuple values at given positions to be used as directory names during write
operations, and directory names as data during read operations.
The key value here is that there is no need to duplicate data values in the directory names and inside the data files.
So only values declared in the parent Tap will be read or written to the underlying file system files. But
fields declared by the Partition will only be read or written to the directory names. That is, the
PartitionTap instance will sink or source the partition fields, plus the parent Tap fields. The partition
fields and parent Tap fields do not need to have common field names.
Note that Hadoop can only sink to directories, and all files in those directories are "part-xxxxx" files.
openWritesThreshold limits the number of open files to be output to. This value defaults to 300 files.
Each time the threshold is exceeded, 10% of the least recently used open files will be closed.
PartitionTap will populate a given partition without regard to case of the values being used. Thus
the resulting paths 2012/June/ and 2012/june/ will likely result in two open files into the same
location. Forcing the case to be consistent with a custom Partition implementation or an upstream
Function is recommended, see ExpressionFunction.
Though Hadoop has no mechanism to prevent simultaneous writes to a directory from multiple jobs, it doesn't mean its safe to do so. Same is true with the PartitionTap. Interleaving writes to a common parent (root) directory across multiple flows will very likely lead to data loss.
BasePartitionTap.Counters, BasePartitionTap.PartitionCollector, BasePartitionTap.PartitionScheme<Config,Input,Output>keepParentOnDelete, OPEN_WRITES_THRESHOLD_DEFAULT, openWritesThreshold, parent, partition, sourcePartitionFiltersCASCADING_SOURCE_PATH| Constructor and Description |
|---|
PartitionTap(Hfs parent,
Partition partition)
|
PartitionTap(Hfs parent,
Partition partition,
int openWritesThreshold)
|
PartitionTap(Hfs parent,
Partition partition,
SinkMode sinkMode)
|
PartitionTap(Hfs parent,
Partition partition,
SinkMode sinkMode,
boolean keepParentOnDelete)
|
PartitionTap(Hfs parent,
Partition partition,
SinkMode sinkMode,
boolean keepParentOnDelete,
int openWritesThreshold)
|
| Modifier and Type | Method and Description |
|---|---|
boolean |
commitResource(Configuration conf) |
protected TupleEntrySchemeCollector |
createTupleEntrySchemeCollector(FlowProcess<? extends Configuration> flowProcess,
Tap parent,
java.lang.String path,
long sequence) |
protected TupleEntrySchemeIterator |
createTupleEntrySchemeIterator(FlowProcess<? extends Configuration> flowProcess,
Tap parent,
java.lang.String path,
RecordReader recordReader) |
protected java.lang.String |
getCurrentIdentifier(FlowProcess<? extends Configuration> flowProcess) |
TupleEntryIterator |
openForRead(FlowProcess<? extends Configuration> flowProcess,
RecordReader input) |
void |
sourceConfInit(FlowProcess<? extends Configuration> flowProcess,
Configuration conf) |
addSourcePartitionFilter, castFileType, createResource, deleteResource, equals, getChildIdentifiers, getChildIdentifiers, getChildIdentifiers, getChildIdentifiers, getChildPartitionIdentifiers, getFilteredPartitionIdentifiers, getIdentifier, getModifiedTime, getOpenWritesThreshold, getParent, getPartition, getSize, getSize, hashCode, isDirectory, isDirectory, openForWrite, prepareResourceForRead, prepareResourceForWrite, resourceExists, rollbackResource, toStringcreateResource, deleteResource, entryStream, entryStream, entryStreamCopy, entryStreamCopy, flowConfInit, getConfigDef, getFullIdentifier, getFullIdentifier, getModifiedTime, getNodeConfigDef, getScheme, getSinkFields, getSinkMode, getSourceFields, getStepConfigDef, getTrace, hasConfigDef, hasNodeConfigDef, hasStepConfigDef, id, isKeep, isReplace, isSink, isSource, isTemporary, isUpdate, openForRead, openForReadUnchecked, openForWrite, outgoingScopeFor, presentSinkFields, presentSourceFields, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, resourceExists, retrieveSinkFields, retrieveSourceFields, setScheme, sinkConfInit, spliterator, splititerator, taps, tupleStream, tupleStream, tupleStreamCopy, tupleStreamCopy@ConstructorProperties(value={"parent","partition"})
public PartitionTap(Hfs parent,
Partition partition)
Hfs Tap as the
base path and default Scheme, and the partition.parent - of type Tappartition - of type Partition@ConstructorProperties(value={"parent","partition","openWritesThreshold"})
public PartitionTap(Hfs parent,
Partition partition,
int openWritesThreshold)
Hfs Tap as the
base path and default Scheme, and the partition.
openWritesThreshold limits the number of open files to be output to.
parent - of type Hfspartition - of type PartitionopenWritesThreshold - of type int@ConstructorProperties(value={"parent","partition","sinkMode"})
public PartitionTap(Hfs parent,
Partition partition,
SinkMode sinkMode)
Hfs Tap as the
base path and default Scheme, and the partition.parent - of type Tappartition - of type StringsinkMode - of type SinkMode@ConstructorProperties(value={"parent","partition","sinkMode","keepParentOnDelete"})
public PartitionTap(Hfs parent,
Partition partition,
SinkMode sinkMode,
boolean keepParentOnDelete)
Hfs Tap as the
base path and default Scheme, and the partition.
keepParentOnDelete, when set to true, prevents the parent Tap from being deleted when BasePartitionTap.deleteResource(Object)
is called, typically an issue when used inside a Cascade.
parent - of type Tappartition - of type PartitionsinkMode - of type SinkModekeepParentOnDelete - of type boolean@ConstructorProperties(value={"parent","partition","sinkMode","keepParentOnDelete","openWritesThreshold"})
public PartitionTap(Hfs parent,
Partition partition,
SinkMode sinkMode,
boolean keepParentOnDelete,
int openWritesThreshold)
Hfs Tap as the
base path and default Scheme, and the partition.
keepParentOnDelete, when set to true, prevents the parent Tap from being deleted when BasePartitionTap.deleteResource(Object)
is called, typically an issue when used inside a Cascade.
openWritesThreshold limits the number of open files to be output to.
parent - of type Tappartition - of type PartitionsinkMode - of type SinkModekeepParentOnDelete - of type booleanopenWritesThreshold - of type intprotected TupleEntrySchemeCollector createTupleEntrySchemeCollector(FlowProcess<? extends Configuration> flowProcess, Tap parent, java.lang.String path, long sequence) throws java.io.IOException
createTupleEntrySchemeCollector in class BasePartitionTap<Configuration,RecordReader,OutputCollector>java.io.IOExceptionprotected TupleEntrySchemeIterator createTupleEntrySchemeIterator(FlowProcess<? extends Configuration> flowProcess, Tap parent, java.lang.String path, RecordReader recordReader) throws java.io.IOException
createTupleEntrySchemeIterator in class BasePartitionTap<Configuration,RecordReader,OutputCollector>java.io.IOExceptionprotected java.lang.String getCurrentIdentifier(FlowProcess<? extends Configuration> flowProcess)
getCurrentIdentifier in class BasePartitionTap<Configuration,RecordReader,OutputCollector>public void sourceConfInit(FlowProcess<? extends Configuration> flowProcess, Configuration conf)
sourceConfInit in class Tap<Configuration,RecordReader,OutputCollector>public TupleEntryIterator openForRead(FlowProcess<? extends Configuration> flowProcess, RecordReader input) throws java.io.IOException
openForRead in class BasePartitionTap<Configuration,RecordReader,OutputCollector>java.io.IOExceptionpublic boolean commitResource(Configuration conf) throws java.io.IOException
commitResource in class BasePartitionTap<Configuration,RecordReader,OutputCollector>java.io.IOExceptionCopyright © 2007-2017 Cascading Maintainers. All Rights Reserved.