cascading.stats.hadoop
Class HadoopStepStats

java.lang.Object
  cascading.stats.CascadingStats
      cascading.stats.FlowStepStats
          cascading.stats.hadoop.HadoopStepStats

All Implemented Interfaces:

Serializable

public abstract class HadoopStepStats
extends FlowStepStats

Class HadoopStepStats provides Hadoop specific statistics and methods to underlying Hadoop facilities.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class cascading.stats.CascadingStats

CascadingStats.Status

Field Summary

static String	COUNTER_TIMEOUT_PROPERTY
static int	TIMEOUT_MAX

Fields inherited from class cascading.stats.CascadingStats

clientState, STATS_STORE_INTERVAL

Constructor Summary

protected

HadoopStepStats(FlowStep<JobConf> flowStep, ClientState clientState)

Method Summary

protected Counters	cachedCounters()
protected Counters	cachedCounters(boolean force)
void	captureDetail() Method captureDetail captures statistics task details and completion events.
void	captureDetail(boolean captureAttempts)
Set<String>	getChildIDs()
Collection	getChildren() Method getChildren returns the children of this HadoopStepStats object.
Collection<String>	getCounterGroups() Method getCounterGroups returns all of the Hadoop counter groups.
Collection<String>	getCounterGroupsMatching(String regex) Method getCounterGroupsMatching returns all the Hadoop counter groups that match the give regex pattern.
Collection<String>	getCountersFor(String group) Method getCountersFor returns the Hadoop counters for the given group.
long	getCounterValue(Enum counter) Method getCounterValue returns the Hadoop counter value for the given counter enum.
long	getCounterValue(String group, String counter) Method getCounterValue returns the Hadoop counter value for the given group and counter name.
abstract JobClient	getJobClient() Method getJobClient returns the Hadoop JobClient managing this Hadoop job.
String	getJobID() Method getJobID returns the Hadoop running job JobID.
float	getMapProgress() Returns the underlying Map tasks progress percentage.
int	getNumMapTasks() Method getNumMapTasks returns the numMapTasks from the Hadoop job file.
int	getNumReduceTasks() Method getNumReduceTasks returns the numReducerTasks from the Hadoop job file.
float	getReduceProgress() Returns the underlying Reduce tasks progress percentage.
abstract RunningJob	getRunningJob() Method getRunningJob returns the Hadoop RunningJob managing this Hadoop job.
String	getStatusURL()
Map<String,HadoopSliceStats>	getTaskStats() Method getTaskStats returns the taskStats of this HadoopStepStats object.
void	recordChildStats() Synchronized to prevent state changes mid record, #stop may be called out of band
protected void	setTaskStats(Map<String,HadoopSliceStats> taskStats)

Methods inherited from class cascading.stats.FlowStepStats

getFlowStep, getID, recordInfo, toString

Methods inherited from class cascading.stats.CascadingStats

cleanup, getCountersFor, getCurrentDuration, getDuration, getFinishedTime, getName, getPendingTime, getRunTime, getStartTime, getStatsString, getStatus, getSubmitTime, getThrowable, isEngaged, isFailed, isFinished, isPending, isRunning, isSkipped, isStarted, isStopped, isSubmitted, isSuccessful, markFailed, markPending, markPendingTime, markRunning, markRunTime, markSkipped, markStarted, markStartedThenRunning, markStartTime, markStartToRunTime, markStopped, markSubmitted, markSubmitTime, markSuccessful, prepare, recordStats, setStatsStoreInterval

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

COUNTER_TIMEOUT_PROPERTY

public static final String COUNTER_TIMEOUT_PROPERTY

See Also:

Constant Field Values

TIMEOUT_MAX

public static final int TIMEOUT_MAX

See Also:

Constant Field Values

Constructor Detail

HadoopStepStats

protected HadoopStepStats(FlowStep<JobConf> flowStep,
                          ClientState clientState)

Method Detail

getTaskStats

public Map<String,HadoopSliceStats> getTaskStats()

Method getTaskStats returns the taskStats of this HadoopStepStats object.

Returns:

the taskStats (type ArrayList) of this HadoopStepStats object.

setTaskStats

protected void setTaskStats(Map<String,HadoopSliceStats> taskStats)

getNumMapTasks

public int getNumMapTasks()

Method getNumMapTasks returns the numMapTasks from the Hadoop job file.

Returns:

the numMapTasks (type int) of this HadoopStepStats object.

getNumReduceTasks

public int getNumReduceTasks()

Method getNumReduceTasks returns the numReducerTasks from the Hadoop job file.

Returns:

the numReducerTasks (type int) of this HadoopStepStats object.

getJobID

public String getJobID()

Method getJobID returns the Hadoop running job JobID.

Returns:

the jobID (type String) of this HadoopStepStats object.

getJobClient

public abstract JobClient getJobClient()

Method getJobClient returns the Hadoop JobClient managing this Hadoop job.

Returns:

the jobClient (type JobClient) of this HadoopStepStats object.

getRunningJob

public abstract RunningJob getRunningJob()

Method getRunningJob returns the Hadoop RunningJob managing this Hadoop job.

Returns:

the runningJob (type RunningJob) of this HadoopStepStats object.

getCounterGroups

public Collection<String> getCounterGroups()

Method getCounterGroups returns all of the Hadoop counter groups.

Specified by:

getCounterGroups in class CascadingStats

Returns:

the counterGroups (type Collection) of this HadoopStepStats object.

getCounterGroupsMatching

public Collection<String> getCounterGroupsMatching(String regex)

Method getCounterGroupsMatching returns all the Hadoop counter groups that match the give regex pattern.

Specified by:

getCounterGroupsMatching in class CascadingStats

Parameters:

regex - of String

Returns:

Collection

getCountersFor

public Collection<String> getCountersFor(String group)

Method getCountersFor returns the Hadoop counters for the given group.

Specified by:

getCountersFor in class CascadingStats

Parameters:

group - of String

Returns:

Collection

getCounterValue

public long getCounterValue(Enum counter)

Method getCounterValue returns the Hadoop counter value for the given counter enum.

Specified by:

getCounterValue in class CascadingStats

Parameters:

counter - of Enum

Returns:

long

getCounterValue

public long getCounterValue(String group,
                            String counter)

Method getCounterValue returns the Hadoop counter value for the given group and counter name.

Specified by:

getCounterValue in class CascadingStats

Parameters:

group - of String

counter - of String

Returns:

long

cachedCounters

protected Counters cachedCounters()

cachedCounters

protected Counters cachedCounters(boolean force)

getMapProgress

public float getMapProgress()

Returns the underlying Map tasks progress percentage.

This method is experimental.

Returns:

float

getReduceProgress

public float getReduceProgress()

Returns the underlying Reduce tasks progress percentage.

This method is experimental.

Returns:

float

getStatusURL

public String getStatusURL()

getChildren

public Collection getChildren()

Method getChildren returns the children of this HadoopStepStats object.

Specified by:

getChildren in class CascadingStats

Returns:

the children (type Collection) of this HadoopStepStats object.

getChildIDs

public Set<String> getChildIDs()

recordChildStats

public void recordChildStats()

Synchronized to prevent state changes mid record, #stop may be called out of band

Specified by:

recordChildStats in class FlowStepStats

captureDetail

public void captureDetail()

Method captureDetail captures statistics task details and completion events.

Specified by:

captureDetail in class CascadingStats

captureDetail

public void captureDetail(boolean captureAttempts)