cascading.scheme

Class Scheme<Config,Input,Output,SourceContext,SinkContext>

java.lang.Object

cascading.scheme.Scheme<Config,Input,Output,SourceContext,SinkContext>

All Implemented Interfaces:

Traceable, java.io.Serializable

Direct Known Subclasses:

BasePartitionTap.PartitionScheme, NullScheme

public abstract class Scheme<Config,Input,Output,SourceContext,SinkContext>
extends java.lang.Object
implements java.io.Serializable, Traceable

A Scheme defines what is stored in a Tap instance by declaring the Tuple field names, and alternately parsing or rendering the incoming or outgoing Tuple stream, respectively.

A Scheme defines the type of resource data will be sourced from or sinked to.

The default sourceFields are Fields.UNKNOWN and the default sinkFields are Fields.ALL.

Any given sourceFields only label the values in the Tuples as they are sourced. It does not necessarily filter the output since a given implementation may choose to collapse values and ignore keys depending on the format.

If the sinkFields are Fields.ALL, the Cascading planner will attempt to resolve the actual field names and make them available via the SinkCall.getOutgoingEntry() method. Sometimes this may not be possible (in the case the Tap.openForWrite(cascading.flow.FlowProcess) method is called from user code directly (without planner intervention).

If the sinkFields are a valid selector, the sink(cascading.flow.FlowProcess, SinkCall) method will only see the fields expected.

Setting the numSinkParts value to 1 (one) attempts to ensure the output resource has only one part. In the case of MapReduce, this is only a suggestion for the Map side, on the Reduce side it does this by setting the number of reducers to the given value. This may affect performance, so be cautioned.

Note that setting numSinkParts does not force the planner to insert a final Reduce operation in the job, so numSinkParts may be ignored entirely if the final job is Map only. To force the Flow to have a final Reduce, add a GroupBy to the assembly before sinking.

See Also:

Serialized Form

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Scheme() Constructor Scheme creates a new Scheme instance.
protected	Scheme(Fields sourceFields) Constructor Scheme creates a new Scheme instance.
protected	Scheme(Fields sourceFields, Fields sinkFields) Constructor Scheme creates a new Scheme instance.
protected	Scheme(Fields sourceFields, Fields sinkFields, int numSinkParts) Constructor Scheme creates a new Scheme instance.
protected	Scheme(Fields sourceFields, int numSinkParts) Constructor Scheme creates a new Scheme instance.

Method Summary

Modifier and Type	Method and Description
boolean	equals(java.lang.Object object)
int	getNumSinkParts() Method getNumSinkParts returns the numSinkParts of this Scheme object.
Fields	getSinkFields() Method getSinkFields returns the sinkFields of this Scheme object.
Fields	getSourceFields() Method getSourceFields returns the sourceFields of this Scheme object.
java.lang.String	getTrace() Method getTrace returns a String that pinpoints the caller that created this instance.
int	hashCode()
boolean	isSink() Method isSink returns true if this Scheme instance can be used as a sink.
boolean	isSource() Method isSource returns true if this Scheme instance can be used as a source.
boolean	isSymmetrical() Method isSymmetrical returns true if the sink fields equal the source fields.
void	presentSinkFields(FlowProcess<? extends Config> flowProcess, Tap tap, Fields fields) Method presentSinkFields is called after the planner is invoked and all fields are resolved.
protected void	presentSinkFieldsInternal(Fields fields)
void	presentSourceFields(FlowProcess<? extends Config> flowProcess, Tap tap, Fields fields) Method presentSourceFields is called after the planner is invoked and all fields are resolved.
protected void	presentSourceFieldsInternal(Fields fields)
Fields	retrieveSinkFields(FlowProcess<? extends Config> flowProcess, Tap tap) Method retrieveSinkFields notifies a Scheme when it is appropriate to dynamically update the fields it sources.
Fields	retrieveSourceFields(FlowProcess<? extends Config> flowProcess, Tap tap) Method retrieveSourceFields notifies a Scheme when it is appropriate to dynamically update the fields it sources.
void	setNumSinkParts(int numSinkParts) Method setNumSinkParts sets the numSinkParts of this Scheme object.
void	setSinkFields(Fields sinkFields) Method setSinkFields sets the sinkFields of this Scheme object.
void	setSourceFields(Fields sourceFields) Method setSourceFields sets the sourceFields of this Scheme object.
abstract void	sink(FlowProcess<? extends Config> flowProcess, SinkCall<SinkContext,Output> sinkCall) Method sink writes out the given Tuple found on SinkCall.getOutgoingEntry() to the SinkCall.getOutput().
void	sinkCleanup(FlowProcess<? extends Config> flowProcess, SinkCall<SinkContext,Output> sinkCall) Method sinkCleanup is used to destroy resources created by sinkPrepare(cascading.flow.FlowProcess, SinkCall).
abstract void	sinkConfInit(FlowProcess<? extends Config> flowProcess, Tap<Config,Input,Output> tap, Config conf) Method sinkInit initializes this instance as a sink.
void	sinkPrepare(FlowProcess<? extends Config> flowProcess, SinkCall<SinkContext,Output> sinkCall) Method sinkPrepare is used to initialize resources needed during each call of sink(cascading.flow.FlowProcess, SinkCall).
Output	sinkWrap(FlowProcess<? extends Config> flowProcess, Output output) Method sinkWrap allows the current Scheme instance to wrap the outgoing Output data source giving the underlying platform direct access to to manage the object in place of the original.
abstract boolean	source(FlowProcess<? extends Config> flowProcess, SourceCall<SourceContext,Input> sourceCall) Method source will read a new "record" or value from SourceCall.getInput() and populate the available Tuple via SourceCall.getIncomingEntry() and return true on success or false if no more values available.
void	sourceCleanup(FlowProcess<? extends Config> flowProcess, SourceCall<SourceContext,Input> sourceCall) Method sourceCleanup is used to destroy resources created by sourcePrepare(cascading.flow.FlowProcess, SourceCall).
abstract void	sourceConfInit(FlowProcess<? extends Config> flowProcess, Tap<Config,Input,Output> tap, Config conf) Method sourceInit initializes this instance as a source.
void	sourcePrepare(FlowProcess<? extends Config> flowProcess, SourceCall<SourceContext,Input> sourceCall) Method sourcePrepare is used to initialize resources needed during each call of source(cascading.flow.FlowProcess, SourceCall).
void	sourceRePrepare(FlowProcess<? extends Config> flowProcess, SourceCall<SourceContext,Input> sourceCall) Method sourceRePrepare is used to re-initialize resources needed during each call of source(cascading.flow.FlowProcess, SourceCall) after the Input object has been changed, if needed.
Input	sourceWrap(FlowProcess<? extends Config> flowProcess, Input input) Method sourceWrap allows the current Scheme instance to wrap the incoming Input data source giving the underlying platform direct access to to manage the object in place of the original.
java.lang.String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

Scheme

protected Scheme()

Constructor Scheme creates a new Scheme instance.

Scheme

protected Scheme(Fields sourceFields)

Constructor Scheme creates a new Scheme instance.

Parameters:

sourceFields - of type Fields

Scheme

protected Scheme(Fields sourceFields,
                 int numSinkParts)

Constructor Scheme creates a new Scheme instance.

Parameters:

sourceFields - of type Fields

numSinkParts - of type int

Scheme

protected Scheme(Fields sourceFields,
                 Fields sinkFields)

Constructor Scheme creates a new Scheme instance.

Parameters:

sourceFields - of type Fields

sinkFields - of type Fields

Scheme

protected Scheme(Fields sourceFields,
                 Fields sinkFields,
                 int numSinkParts)

Constructor Scheme creates a new Scheme instance.

Parameters:

sourceFields - of type Fields

sinkFields - of type Fields

numSinkParts - of type int

Method Detail

getSinkFields

public Fields getSinkFields()

Method getSinkFields returns the sinkFields of this Scheme object.

Returns:

the sinkFields (type Fields) of this Scheme object.

setSinkFields

public void setSinkFields(Fields sinkFields)

Method setSinkFields sets the sinkFields of this Scheme object.

Parameters:

sinkFields - the sinkFields of this Scheme object.

getSourceFields

public Fields getSourceFields()

Method getSourceFields returns the sourceFields of this Scheme object.

Returns:

the sourceFields (type Fields) of this Scheme object.

setSourceFields

public void setSourceFields(Fields sourceFields)

Method setSourceFields sets the sourceFields of this Scheme object.

Parameters:

sourceFields - the sourceFields of this Scheme object.

getNumSinkParts

public int getNumSinkParts()

Method getNumSinkParts returns the numSinkParts of this Scheme object.

Returns:

the numSinkParts (type int) of this Scheme object.

setNumSinkParts

public void setNumSinkParts(int numSinkParts)

Method setNumSinkParts sets the numSinkParts of this Scheme object.

Parameters:

numSinkParts - the numSinkParts of this Scheme object.

getTrace

public java.lang.String getTrace()

Description copied from interface: Traceable

Method getTrace returns a String that pinpoints the caller that created this instance.

Specified by:

getTrace in interface Traceable

Returns:

String

isSymmetrical

public boolean isSymmetrical()

Method isSymmetrical returns true if the sink fields equal the source fields. That is, this scheme sources the same fields as it sinks.

Returns:

the symmetrical (type boolean) of this Scheme object.

isSource

public boolean isSource()

Method isSource returns true if this Scheme instance can be used as a source.

Returns:

boolean

isSink

public boolean isSink()

Method isSink returns true if this Scheme instance can be used as a sink.

Returns:

boolean

retrieveSourceFields

public Fields retrieveSourceFields(FlowProcess<? extends Config> flowProcess,
                                   Tap tap)

Method retrieveSourceFields notifies a Scheme when it is appropriate to dynamically update the fields it sources. By default the current declared fields are returned.

The FlowProcess presents all known properties resolved by the current planner.

The tap instance is the parent Tap for this Scheme instance.

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

Returns:

Fields

presentSourceFields

public void presentSourceFields(FlowProcess<? extends Config> flowProcess,
                                Tap tap,
                                Fields fields)

Method presentSourceFields is called after the planner is invoked and all fields are resolved. This method presents to the Scheme the actual source fields after any planner intervention.

This method is called after retrieveSourceFields(cascading.flow.FlowProcess, cascading.tap.Tap).

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

fields - of type Fields

presentSourceFieldsInternal

protected void presentSourceFieldsInternal(Fields fields)

retrieveSinkFields

public Fields retrieveSinkFields(FlowProcess<? extends Config> flowProcess,
                                 Tap tap)

Method retrieveSinkFields notifies a Scheme when it is appropriate to dynamically update the fields it sources. By default the current declared fields are returned.

The FlowProcess presents all known properties resolved by the current planner.

The tap instance is the parent Tap for this Scheme instance.

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

Returns:

Fields

presentSinkFields

public void presentSinkFields(FlowProcess<? extends Config> flowProcess,
                              Tap tap,
                              Fields fields)

Method presentSinkFields is called after the planner is invoked and all fields are resolved. This method presents to the Scheme the actual source fields after any planner intervention.

This method is called after retrieveSinkFields(cascading.flow.FlowProcess, cascading.tap.Tap).

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

fields - of type Fields

presentSinkFieldsInternal

protected void presentSinkFieldsInternal(Fields fields)

sourceConfInit

public abstract void sourceConfInit(FlowProcess<? extends Config> flowProcess,
                                    Tap<Config,Input,Output> tap,
                                    Config conf)

Method sourceInit initializes this instance as a source.

This method is executed client side as a means to provide necessary configuration parameters used by the underlying platform.

It is not intended to initialize resources that would be necessary during the execution of this class, like a "formatter" or "parser".

See sourcePrepare(cascading.flow.FlowProcess, SourceCall) if resources much be initialized before use. And sourceCleanup(cascading.flow.FlowProcess, SourceCall) if resources must be destroyed after use.

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

conf - of type Config

sinkConfInit

public abstract void sinkConfInit(FlowProcess<? extends Config> flowProcess,
                                  Tap<Config,Input,Output> tap,
                                  Config conf)

Method sinkInit initializes this instance as a sink.

This method is executed client side as a means to provide necessary configuration parameters used by the underlying platform.

It is not intended to initialize resources that would be necessary during the execution of this class, like a "formatter" or "parser".

See sinkPrepare(cascading.flow.FlowProcess, SinkCall) if resources much be initialized before use. And sinkCleanup(cascading.flow.FlowProcess, SinkCall) if resources must be destroyed after use.

Parameters:

flowProcess - of type FlowProcess

tap - of type Tap

conf - of type Config

sourceWrap

public Input sourceWrap(FlowProcess<? extends Config> flowProcess,
                        Input input)
                 throws java.io.IOException

Method sourceWrap allows the current Scheme instance to wrap the incoming Input data source giving the underlying platform direct access to to manage the object in place of the original.

If the Input is an InputStream, the stream can be decompressed by wrapping in an appropriate de-compressor InputStream.

Parameters:

flowProcess - of type FlowProcess

input - the Input provided by the platform

Returns:

the same or an instance of the Input type wrapping the given parameter

Throws:

java.io.IOException - if unable to wrap the parameter

sourcePrepare

public void sourcePrepare(FlowProcess<? extends Config> flowProcess,
                          SourceCall<SourceContext,Input> sourceCall)
                   throws java.io.IOException

Method sourcePrepare is used to initialize resources needed during each call of source(cascading.flow.FlowProcess, SourceCall).

This method is guaranteed to be called once before the first invocation of source(FlowProcess, SourceCall).

Be sure to place any initialized objects in the SourceContext so each instance will remain thread-safe.

Parameters:

flowProcess - of type FlowProcess

sourceCall - of type SourceCall

Throws:

java.io.IOException

sourceRePrepare

public void sourceRePrepare(FlowProcess<? extends Config> flowProcess,
                            SourceCall<SourceContext,Input> sourceCall)
                     throws java.io.IOException

Method sourceRePrepare is used to re-initialize resources needed during each call of source(cascading.flow.FlowProcess, SourceCall) after the Input object has been changed, if needed.

This method may be called zero or more times. Note sourcePrepare(FlowProcess, SourceCall) will always be called before any source(FlowProcess, SourceCall) invocation.

Parameters:

flowProcess - of type FlowProcess

sourceCall - of type SourceCall

Throws:

java.io.IOException

source

public abstract boolean source(FlowProcess<? extends Config> flowProcess,
                               SourceCall<SourceContext,Input> sourceCall)
                        throws java.io.IOException

Method source will read a new "record" or value from SourceCall.getInput() and populate the available Tuple via SourceCall.getIncomingEntry() and return true on success or false if no more values available.

It's ok to set a new Tuple instance on the incomingEntry TupleEntry, or to simply re-use the existing instance.

Note this is only time it is safe to modify a Tuple instance handed over via a method call.

This method may optionally throw a TapException if it cannot process a particular instance of data. If the payload Tuple is set on the TapException, that Tuple will be written to any applicable failure trap Tap.

Parameters:

flowProcess - of type FlowProcess

sourceCall - of SourceCall

Returns:

returns true when a Tuple was successfully read

Throws:

java.io.IOException

sourceCleanup

public void sourceCleanup(FlowProcess<? extends Config> flowProcess,
                          SourceCall<SourceContext,Input> sourceCall)
                   throws java.io.IOException

Method sourceCleanup is used to destroy resources created by sourcePrepare(cascading.flow.FlowProcess, SourceCall).

Parameters:

flowProcess - of Process

sourceCall - of type SourceCall

Throws:

java.io.IOException

sinkWrap

public Output sinkWrap(FlowProcess<? extends Config> flowProcess,
                       Output output)
                throws java.io.IOException

Method sinkWrap allows the current Scheme instance to wrap the outgoing Output data source giving the underlying platform direct access to to manage the object in place of the original.

If the Output is an OutputStream, the stream can be compressed by wrapping in an appropriate compressor OutputStream.

Parameters:

flowProcess - of type FlowProcess

output - the Output provided by the platform

Returns:

the same or an instance of the Input type wrapping the given parameter

Throws:

java.io.IOException - if unable to wrap the parameter

sinkPrepare

public void sinkPrepare(FlowProcess<? extends Config> flowProcess,
                        SinkCall<SinkContext,Output> sinkCall)
                 throws java.io.IOException

Method sinkPrepare is used to initialize resources needed during each call of sink(cascading.flow.FlowProcess, SinkCall).

This method is guaranteed to be called once before the first invocation of sink(FlowProcess, SinkCall).

Be sure to place any initialized objects in the SinkContext so each instance will remain threadsafe.

Parameters:

flowProcess - of type FlowProcess

sinkCall - of type SinkCall

Throws:

java.io.IOException

sink

public abstract void sink(FlowProcess<? extends Config> flowProcess,
                          SinkCall<SinkContext,Output> sinkCall)
                   throws java.io.IOException

Method sink writes out the given Tuple found on SinkCall.getOutgoingEntry() to the SinkCall.getOutput().

This method may optionally throw a TapException if it cannot process a particular instance of data. If the payload Tuple is set on the TapException, that Tuple will be written to any applicable failure trap Tap. If not set, the incoming Tuple will be written instead.

Parameters:

flowProcess - of Process

sinkCall - of SinkCall

Throws:

java.io.IOException

sinkCleanup

public void sinkCleanup(FlowProcess<? extends Config> flowProcess,
                        SinkCall<SinkContext,Output> sinkCall)
                 throws java.io.IOException

Method sinkCleanup is used to destroy resources created by sinkPrepare(cascading.flow.FlowProcess, SinkCall).

Parameters:

flowProcess - of type FlowProcess

sinkCall - of type SinkCall

Throws:

java.io.IOException

equals

public boolean equals(java.lang.Object object)

Overrides:

equals in class java.lang.Object

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object