cascading.pipe.assembly
Class AverageBy

java.lang.Object
  cascading.pipe.Pipe
      cascading.pipe.SubAssembly
          cascading.pipe.assembly.AggregateBy
              cascading.pipe.assembly.AverageBy

All Implemented Interfaces:

FlowElement, Traceable, Serializable

public class AverageBy
extends AggregateBy

Class AverageBy is used to average values associated with duplicate keys in a tuple stream.

Typically finding the average value in a tuple stream relies on a GroupBy and a Average Aggregator operation.

If the given averageFields has an associated type, this type will be used to coerce the resulting average value, otherwise the result will be a Double.

If include is AverageBy.Include.NO_NULLS, null values will not be included in the average (converted to zero). By default (and for backwards compatibility) null values are included, AverageBy.Include.ALL.

This SubAssembly uses the AverageBy.AveragePartials AggregateBy.Functor and private AverageBy.AverageFinal Aggregator to count and sum as many field values before the GroupBy operator to reduce IO over the network.

This strategy is similar to using combiners, except no sorting or serialization is invoked and results in a much simpler mechanism.

The threshold value tells the underlying AveragePartials functions how many unique key sums and counts to accumulate in the LRU cache, before emitting the least recently used entry. This accumulation happens map-side, and thus is bounded by the size of your map task JVM and the typical size of each group key.

By default, either the value of AggregateByProps.AGGREGATE_BY_CAPACITY System property or AggregateByProps.AGGREGATE_BY_DEFAULT_CAPACITY will be used.

See Also:

AggregateBy, Serialized Form

Nested Class Summary

static class	AverageBy.AverageFinal Class AverageFinal is used to finalize the average operation on the Reduce side of the process.
static class	AverageBy.AveragePartials Class AveragePartials is a AggregateBy.Functor that is used to count and sum observed duplicates from the tuple stream.
static class	AverageBy.Include

Nested classes/interfaces inherited from class cascading.pipe.assembly.AggregateBy

AggregateBy.Cache, AggregateBy.CompositeFunction, AggregateBy.Flush, AggregateBy.Functor

Field Summary

static int

DEFAULT_THRESHOLD Deprecated.

Fields inherited from class cascading.pipe.assembly.AggregateBy

AGGREGATE_BY_THRESHOLD, USE_DEFAULT_THRESHOLD

Fields inherited from class cascading.pipe.Pipe

configDef, parent, stepConfigDef

Constructor Summary

AverageBy(Fields valueField, Fields averageField)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Fields valueField, Fields averageField, AverageBy.Include include)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include, int threshold)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField, int threshold)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include, int threshold)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField, int threshold)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include, int threshold)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe[] pipes, Fields groupingFields, Fields valueField, Fields averageField, int threshold)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField, AverageBy.Include include, int threshold)
Constructor AverageBy creates a new AverageBy instance.

AverageBy(String name, Pipe pipe, Fields groupingFields, Fields valueField, Fields averageField, int threshold)
Constructor AverageBy creates a new AverageBy instance.

Method Summary

Methods inherited from class cascading.pipe.assembly.AggregateBy

getAggregators, getArgumentFields, getCapacity, getFieldDeclarations, getFunctors, getGroupBy, getGroupingFields, getThreshold, initialize, initialize, verify

Methods inherited from class cascading.pipe.SubAssembly

getName, getPrevious, getTailNames, getTails, setPrevious, setTails, unwind

Methods inherited from class cascading.pipe.Pipe

equals, getConfigDef, getHeads, getParent, getStepConfigDef, getTrace, hasConfigDef, hashCode, hasStepConfigDef, id, isEquivalentTo, named, names, outgoingScopeFor, pipes, print, printInternal, resolveIncomingOperationArgumentFields, resolveIncomingOperationPassThroughFields, setParent, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_THRESHOLD

@Deprecated
public static final int DEFAULT_THRESHOLD

Deprecated.

DEFAULT_THRESHOLD

See Also:

Constant Field Values

Constructor Detail

AverageBy

@ConstructorProperties(value={"valueField","averageField"})
public AverageBy(Fields valueField,
                                            Fields averageField)

Constructor AverageBy creates a new AverageBy instance. Use this constructor when used with a AggregateBy instance.

Parameters:

valueField - of type Fields

averageField - of type Fields

AverageBy

@ConstructorProperties(value={"valueField","averageField","include"})
public AverageBy(Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include)

Constructor AverageBy creates a new AverageBy instance. Use this constructor when used with a AggregateBy instance.

Parameters:

valueField - of type Fields

averageField - of type Fields

include - of type boolean

AverageBy

@ConstructorProperties(value={"pipe","groupingFields","valueField","averageField"})
public AverageBy(Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

AverageBy

@ConstructorProperties(value={"pipe","groupingFields","valueField","averageField","threshold"})
public AverageBy(Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

threshold - of type int

AverageBy

@ConstructorProperties(value={"name","pipe","groupingFields","valueField","averageField"})
public AverageBy(String name,
                                            Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

AverageBy

@ConstructorProperties(value={"name","pipe","groupingFields","valueField","averageField","threshold"})
public AverageBy(String name,
                                            Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

threshold - of type int

AverageBy

@ConstructorProperties(value={"pipes","groupingFields","valueField","averageField"})
public AverageBy(Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

AverageBy

@ConstructorProperties(value={"pipes","groupingFields","valueField","averageField","threshold"})
public AverageBy(Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

threshold - of type int

AverageBy

@ConstructorProperties(value={"name","pipes","groupingFields","valueField","averageField"})
public AverageBy(String name,
                                            Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

AverageBy

@ConstructorProperties(value={"name","pipes","groupingFields","valueField","averageField","threshold"})
public AverageBy(String name,
                                            Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

threshold - of type int

AverageBy

@ConstructorProperties(value={"pipe","groupingFields","valueField","averageField","include"})
public AverageBy(Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

AverageBy

@ConstructorProperties(value={"pipe","groupingFields","valueField","averageField","include","threshold"})
public AverageBy(Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

threshold - of type int

AverageBy

@ConstructorProperties(value={"name","pipe","groupingFields","valueField","averageField","include"})
public AverageBy(String name,
                                            Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

AverageBy

@ConstructorProperties(value={"name","pipe","groupingFields","valueField","averageField","include","threshold"})
public AverageBy(String name,
                                            Pipe pipe,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipe - of type Pipe

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

threshold - of type int

AverageBy

@ConstructorProperties(value={"pipes","groupingFields","valueField","averageField","include"})
public AverageBy(Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

AverageBy

@ConstructorProperties(value={"pipes","groupingFields","valueField","averageField","include","threshold"})
public AverageBy(Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

threshold - of type int

AverageBy

@ConstructorProperties(value={"name","pipes","groupingFields","valueField","averageField","include"})
public AverageBy(String name,
                                            Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

AverageBy

@ConstructorProperties(value={"name","pipes","groupingFields","valueField","averageField","include","threshold"})
public AverageBy(String name,
                                            Pipe[] pipes,
                                            Fields groupingFields,
                                            Fields valueField,
                                            Fields averageField,
                                            AverageBy.Include include,
                                            int threshold)

Constructor AverageBy creates a new AverageBy instance.

Parameters:

name - of type String

pipes - of type Pipe[]

groupingFields - of type Fields

valueField - of type Fields

averageField - of type Fields

include - of type boolean

threshold - of type int