cascading.tuple.collect
Class SpillableTupleMap

java.lang.Object
  extended by java.util.AbstractMap<K,V>
      extended by java.util.HashMap<Tuple,Collection<Tuple>>
          extended by cascading.tuple.collect.SpillableTupleMap
All Implemented Interfaces:
Spillable, Serializable, Cloneable, Map<Tuple,Collection<Tuple>>

public abstract class SpillableTupleMap
extends HashMap<Tuple,Collection<Tuple>>
implements Spillable

SpillableTupleMap is a HashMap that will allow for multiple values per key, and if the number of values for a given key reach a specific threshold, they will be spilled to disk using a SpillableTupleList instance. Only values are spilled, keys are not spilled and too many keys can result in a OutOfMemoryError.

The SpillableProps.MAP_THRESHOLD value sets the number of total values that this map will strive to keep in memory regardless of the number of keys. This is achieved by dynamically calculating the threshold used by each child SpillableTupleList instance using threshold = Min( list_threshold, map_threshold / current_num_keys ) .

To set the list threshold, see SpillableProps fluent helper class.

This class is used by the HashJoin pipe, to set properties specific to a given join instance, see the Pipe.getConfigDef() method.

See Also:
HadoopSpillableTupleMap, Serialized Form

Nested Class Summary
 
Nested classes/interfaces inherited from class java.util.AbstractMap
AbstractMap.SimpleEntry<K,V>, AbstractMap.SimpleImmutableEntry<K,V>
 
Nested classes/interfaces inherited from interface cascading.tuple.collect.Spillable
Spillable.SpillListener, Spillable.SpillStrategy
 
Field Summary
static String MAP_CAPACITY
          Deprecated. 
static String MAP_LOADFACTOR
          Deprecated. 
static String MAP_THRESHOLD
          Deprecated. 
 
Constructor Summary
SpillableTupleMap(int initialCapacity, float loadFactor, int mapThreshold, int initListThreshold)
           
SpillableTupleMap(int mapThreshold, int initListThreshold)
           
 
Method Summary
protected abstract  Collection<Tuple> createTupleCollection(Tuple object)
           
 Collection<Tuple> get(Object object)
           
 Tuple getGrouping()
           
 int getInitListThreshold()
           
static int getMapCapacity(FlowProcess flowProcess, int defaultValue)
           
static float getMapLoadFactor(FlowProcess flowProcess, float defaultValue)
           
protected  int getMapThreshold()
           
static int getMapThreshold(FlowProcess flowProcess, int defaultValue)
           
 Spillable.SpillListener getSpillListener()
           
 void setGrouping(Tuple group)
           
 void setSpillListener(Spillable.SpillListener spillListener)
           
 void setSpillStrategy(Spillable.SpillStrategy spillStrategy)
           
 int spillCount()
          The number of times this container has spilled data to disk.
 
Methods inherited from class java.util.HashMap
clear, clone, containsKey, containsValue, entrySet, isEmpty, keySet, put, putAll, remove, size, values
 
Methods inherited from class java.util.AbstractMap
equals, hashCode, toString
 
Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, wait, wait, wait
 
Methods inherited from interface java.util.Map
equals, hashCode
 

Field Detail

MAP_THRESHOLD

@Deprecated
public static final String MAP_THRESHOLD
Deprecated. 
The total number of tuple values (not keys) to attempt to keep in memory.

See Also:
Constant Field Values

MAP_CAPACITY

@Deprecated
public static final String MAP_CAPACITY
Deprecated. 
The initial hash map capacity.

See Also:
HashMap, Constant Field Values

MAP_LOADFACTOR

@Deprecated
public static final String MAP_LOADFACTOR
Deprecated. 
The initial hash map load factor.

See Also:
HashMap, Constant Field Values
Constructor Detail

SpillableTupleMap

public SpillableTupleMap(int mapThreshold,
                         int initListThreshold)

SpillableTupleMap

public SpillableTupleMap(int initialCapacity,
                         float loadFactor,
                         int mapThreshold,
                         int initListThreshold)
Method Detail

getMapThreshold

public static int getMapThreshold(FlowProcess flowProcess,
                                  int defaultValue)

getMapCapacity

public static int getMapCapacity(FlowProcess flowProcess,
                                 int defaultValue)

getMapLoadFactor

public static float getMapLoadFactor(FlowProcess flowProcess,
                                     float defaultValue)

getMapThreshold

protected int getMapThreshold()

getInitListThreshold

public int getInitListThreshold()

get

public Collection<Tuple> get(Object object)
Specified by:
get in interface Map<Tuple,Collection<Tuple>>
Overrides:
get in class HashMap<Tuple,Collection<Tuple>>

createTupleCollection

protected abstract Collection<Tuple> createTupleCollection(Tuple object)

setGrouping

public void setGrouping(Tuple group)
Specified by:
setGrouping in interface Spillable

getGrouping

public Tuple getGrouping()
Specified by:
getGrouping in interface Spillable

setSpillStrategy

public void setSpillStrategy(Spillable.SpillStrategy spillStrategy)
Specified by:
setSpillStrategy in interface Spillable

spillCount

public int spillCount()
Description copied from interface: Spillable
The number of times this container has spilled data to disk.

Specified by:
spillCount in interface Spillable
Returns:
in int

getSpillListener

public Spillable.SpillListener getSpillListener()

setSpillListener

public void setSpillListener(Spillable.SpillListener spillListener)
Specified by:
setSpillListener in interface Spillable


Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.