public class HadoopSpillableTupleList extends SpillableTupleList
Iterable
object that can store an unlimited number of Tuple
instances by spilling
excess to a temporary disk file.
Spills will automatically be compressed using the defaultCodecs
values. To disable compression or
change the codecs, see SpillableProps.SPILL_COMPRESS
and SpillableProps.SPILL_CODECS
.
It is recommended to add Lzo if available.
"org.apache.hadoop.io.compress.LzoCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec"
Spillable.SpillListener, Spillable.SpillStrategy
Modifier and Type | Field and Description |
---|---|
static String |
defaultCodecs |
Constructor and Description |
---|
HadoopSpillableTupleList(int threshold,
CompressionCodec codec,
Configuration configuration)
Constructor SpillableTupleList creates a new SpillableTupleList instance using the given threshold value, and
the first available compression codec, if any.
|
HadoopSpillableTupleList(int threshold,
TupleSerialization tupleSerialization,
CompressionCodec codec) |
Modifier and Type | Method and Description |
---|---|
protected TupleInputStream |
createTupleInputStream(File file) |
protected TupleOutputStream |
createTupleOutputStream(File file) |
static CompressionCodec |
getCodec(FlowProcess<? extends Configuration> flowProcess,
String defaultCodecs) |
add, addAll, clear, contains, containsAll, getCodecClass, getGrouping, getThreshold, isEmpty, iterator, remove, removeAll, retainAll, setGrouping, setSpillListener, setSpillStrategy, size, spillCount, toArray, toArray
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
equals, hashCode
public static final String defaultCodecs
public HadoopSpillableTupleList(int threshold, CompressionCodec codec, Configuration configuration)
threshold
- of type longcodec
- of type CompressionCodecpublic HadoopSpillableTupleList(int threshold, TupleSerialization tupleSerialization, CompressionCodec codec)
public static CompressionCodec getCodec(FlowProcess<? extends Configuration> flowProcess, String defaultCodecs)
protected TupleOutputStream createTupleOutputStream(File file)
createTupleOutputStream
in class SpillableTupleList
protected TupleInputStream createTupleInputStream(File file)
createTupleInputStream
in class SpillableTupleList
Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.