cascading.flow.hadoop
Class HadoopFlowProcess

java.lang.Object
  cascading.flow.FlowProcess<JobConf>
      cascading.flow.hadoop.HadoopFlowProcess

public class HadoopFlowProcess
extends FlowProcess<JobConf>

Class HadoopFlowProcess is an implementation of FlowProcess for Hadoop. Use this interface to get direct access to the Hadoop JobConf and Reporter interfaces.

Be warned that coupling to this implementation will cause custom Operations to fail if they are executed on a system other than Hadoop.

See Also:

FlowSession, JobConf, Reporter

Nested Class Summary

Nested classes/interfaces inherited from class cascading.flow.FlowProcess

FlowProcess.NullFlowProcess

Field Summary

Fields inherited from class cascading.flow.FlowProcess

NULL

Constructor Summary

HadoopFlowProcess()

HadoopFlowProcess(FlowSession flowSession, JobConf jobConf)

HadoopFlowProcess(FlowSession flowSession, JobConf jobConf, boolean isMapper)
Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.

HadoopFlowProcess(HadoopFlowProcess flowProcess, JobConf jobConf)

HadoopFlowProcess(JobConf jobConf)

Method Summary

JobConf	copyConfig(JobConf jobConf)
FlowProcess	copyWith(JobConf jobConf)
Map<String,String>	diffConfigIntoMap(JobConf defaultConfig, JobConf updatedConfig)
JobConf	getConfigCopy()
int	getCurrentNumMappers()
int	getCurrentNumReducers()
int	getCurrentSliceNum() Method getCurrentTaskNum returns the task number of this task.
JobConf	getJobConf() Method getJobConf returns the jobConf of this HadoopFlowProcess object.
int	getNumProcessSlices()
OutputCollector	getOutputCollector()
Object	getProperty(String key)
Collection<String>	getPropertyKeys()
Reporter	getReporter() Method getReporter returns the reporter of this HadoopFlowProcess object.
void	increment(Enum counter, long amount)
void	increment(String group, String counter, long amount)
boolean	isCounterStatusInitialized()
boolean	isMapper() Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper.
void	keepAlive()
JobConf	mergeMapIntoConfig(JobConf defaultConfig, Map<String,String> map)
Object	newInstance(String className)
TupleEntryCollector	openSystemIntermediateForWrite()
TupleEntryIterator	openTapForRead(Tap tap)
TupleEntryCollector	openTapForWrite(Tap tap)
TupleEntryCollector	openTrapForWrite(Tap trap)
void	setOutputCollector(OutputCollector outputCollector)
void	setReporter(Reporter reporter) Method setReporter sets the reporter of this HadoopFlowProcess object.
void	setStatus(String status)

Methods inherited from class cascading.flow.FlowProcess

getCurrentSession, getID, getIntegerProperty, getStringProperty, setCurrentSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopFlowProcess

public HadoopFlowProcess()

HadoopFlowProcess

public HadoopFlowProcess(JobConf jobConf)

HadoopFlowProcess

public HadoopFlowProcess(FlowSession flowSession,
                         JobConf jobConf)

HadoopFlowProcess

public HadoopFlowProcess(FlowSession flowSession,
                         JobConf jobConf,
                         boolean isMapper)

Constructor HadoopFlowProcess creates a new HadoopFlowProcess instance.

Parameters:

flowSession - of type FlowSession

jobConf - of type JobConf

HadoopFlowProcess

public HadoopFlowProcess(HadoopFlowProcess flowProcess,
                         JobConf jobConf)

Method Detail

copyWith

public FlowProcess copyWith(JobConf jobConf)

Specified by:

copyWith in class FlowProcess<JobConf>

getJobConf

public JobConf getJobConf()

Method getJobConf returns the jobConf of this HadoopFlowProcess object.

Returns:

the jobConf (type JobConf) of this HadoopFlowProcess object.

getConfigCopy

public JobConf getConfigCopy()

Specified by:

getConfigCopy in class FlowProcess<JobConf>

isMapper

public boolean isMapper()

Method isMapper returns true if this part of the FlowProcess is a MapReduce mapper. If false, it is a reducer.

Returns:

boolean

getCurrentNumMappers

public int getCurrentNumMappers()

getCurrentNumReducers

public int getCurrentNumReducers()

getCurrentSliceNum

public int getCurrentSliceNum()

Method getCurrentTaskNum returns the task number of this task. Task 0 is the first task.

Specified by:

getCurrentSliceNum in class FlowProcess<JobConf>

Returns:

int

getNumProcessSlices

public int getNumProcessSlices()

Specified by:

getNumProcessSlices in class FlowProcess<JobConf>

setReporter

public void setReporter(Reporter reporter)

Method setReporter sets the reporter of this HadoopFlowProcess object.

Parameters:

reporter - the reporter of this HadoopFlowProcess object.

getReporter

public Reporter getReporter()

Method getReporter returns the reporter of this HadoopFlowProcess object.

Returns:

the reporter (type Reporter) of this HadoopFlowProcess object.

setOutputCollector

public void setOutputCollector(OutputCollector outputCollector)

getOutputCollector

public OutputCollector getOutputCollector()

getProperty

public Object getProperty(String key)

Specified by:

getProperty in class FlowProcess<JobConf>

getPropertyKeys

public Collection<String> getPropertyKeys()

Specified by:

getPropertyKeys in class FlowProcess<JobConf>

newInstance

public Object newInstance(String className)

Specified by:

newInstance in class FlowProcess<JobConf>

keepAlive

public void keepAlive()

Specified by:

keepAlive in class FlowProcess<JobConf>

increment

public void increment(Enum counter,
                      long amount)

Specified by:

increment in class FlowProcess<JobConf>

increment

public void increment(String group,
                      String counter,
                      long amount)

Specified by:

increment in class FlowProcess<JobConf>

setStatus

public void setStatus(String status)

Specified by:

setStatus in class FlowProcess<JobConf>

isCounterStatusInitialized

public boolean isCounterStatusInitialized()

Specified by:

isCounterStatusInitialized in class FlowProcess<JobConf>

openTapForRead

public TupleEntryIterator openTapForRead(Tap tap)
                                  throws IOException

Specified by:

openTapForRead in class FlowProcess<JobConf>

Throws:

IOException

openTapForWrite

public TupleEntryCollector openTapForWrite(Tap tap)
                                    throws IOException

Specified by:

openTapForWrite in class FlowProcess<JobConf>

Throws:

IOException

openTrapForWrite

public TupleEntryCollector openTrapForWrite(Tap trap)
                                     throws IOException

Specified by:

openTrapForWrite in class FlowProcess<JobConf>

Throws:

IOException

openSystemIntermediateForWrite

public TupleEntryCollector openSystemIntermediateForWrite()
                                                   throws IOException

Specified by:

openSystemIntermediateForWrite in class FlowProcess<JobConf>

Throws:

IOException

copyConfig

public JobConf copyConfig(JobConf jobConf)

Specified by:

copyConfig in class FlowProcess<JobConf>

diffConfigIntoMap

public Map<String,String> diffConfigIntoMap(JobConf defaultConfig,
                                            JobConf updatedConfig)

Specified by:

diffConfigIntoMap in class FlowProcess<JobConf>

mergeMapIntoConfig

public JobConf mergeMapIntoConfig(JobConf defaultConfig,
                                  Map<String,String> map)

Specified by:

mergeMapIntoConfig in class FlowProcess<JobConf>