cascading.flow.planner

Class FlowPlanner<F extends BaseFlow,Config>

java.lang.Object

cascading.flow.planner.FlowPlanner<F,Config>

public abstract class FlowPlanner<F extends BaseFlow,Config>
extends java.lang.Object

Class FlowPlanner is the base class for all planner implementations.

This planner support tracing execution of each rule. See the appropriate properties on this class to enable.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	FlowPlanner.IdentityElementFactgory
class	FlowPlanner.TempTapElementFactory

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	checkpointTapRootPath
protected AssertionLevel	defaultAssertionLevel Field assertionLevel
protected DebugLevel	defaultDebugLevel Field debugLevel
protected java.util.Map<java.lang.Object,java.lang.Object>	defaultProperties Field properties
static java.lang.String	TRACE_PLAN_PATH Enables the planner to write out basic planner information including the initial element-graph, completed element-graph, and the completed step-graph dot files.
static java.lang.String	TRACE_PLAN_TRANSFORM_PATH Enables the planner to write out detail level planner information for each rule, including recursive transforms.
static java.lang.String	TRACE_STATS_PATH Enables the planner to write out planner statistics for each planner phase and rule.

Constructor Summary

Constructors

Constructor and Description
FlowPlanner()

Method Summary

Modifier and Type	Method and Description
F	buildFlow(FlowDef flowDef, RuleRegistrySet ruleRegistrySet)
void	configRuleRegistryDefaults(RuleRegistry ruleRegistry)
protected abstract F	createFlow(FlowDef flowDef)
protected FlowElementGraph	createFlowElementGraph(FlowDef flowDef, Pipe[] flowTails)
protected AssertionLevel	getAssertionLevel(FlowDef flowDef)
protected DebugLevel	getDebugLevel(FlowDef flowDef)
abstract Config	getDefaultConfig()
java.util.Map<java.lang.Object,java.lang.Object>	getDefaultProperties()
FlowNodeFactory	getFlowNodeFactory()
abstract FlowStepFactory<Config>	getFlowStepFactory()
abstract PlannerInfo	getPlannerInfo(java.lang.String name)
abstract PlatformInfo	getPlatformInfo()
protected PlannerException	handleExceptionDuringPlanning(FlowDef flowDef, java.lang.Exception exception, FlowElementGraph flowElementGraph)
void	initialize(FlowConnector flowConnector, java.util.Map<java.lang.Object,java.lang.Object> properties)
protected Tap	makeTempTap(java.lang.String name)
protected abstract Tap	makeTempTap(java.lang.String prefix, java.lang.String name)
protected Pipe[]	resolveAssemblyPlanners(FlowDef flowDef, Flow flow, Pipe[] pipes)
protected Pipe[]	resolveTails(FlowDef flowDef, F flow)
protected void	verifyAllTaps(FlowDef flowDef)
protected void	verifyAssembly(FlowDef flowDef, Pipe[] tails)
protected void	verifyCheckpoints(FlowDef flowDef, Pipe[] flowTails)
protected void	verifyPipeAssemblyEndPoints(FlowDef flowDef, Pipe[] flowTails) Method verifyEndPoints verifies
java.lang.Exception	verifyResult(RuleResult ruleResult) If there are rules for a given ProcessLevel on the current platform there must be sub-graphs partitioned at that level.
protected void	verifyResultInternal(RuleResult ruleResult)
protected void	verifySourceNotSinks(java.util.Map<java.lang.String,Tap> sources, java.util.Map<java.lang.String,Tap> sinks)
protected void	verifyTaps(java.util.Map<java.lang.String,Tap> taps, boolean areSources, boolean mayNotBeEmpty) Method verifyTaps ...
protected void	verifyTraps(FlowDef flowDef, Pipe[] flowTails)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TRACE_PLAN_PATH

public static final java.lang.String TRACE_PLAN_PATH

Enables the planner to write out basic planner information including the initial element-graph, completed element-graph, and the completed step-graph dot files.

See Also:

Constant Field Values

TRACE_PLAN_TRANSFORM_PATH

public static final java.lang.String TRACE_PLAN_TRANSFORM_PATH

Enables the planner to write out detail level planner information for each rule, including recursive transforms.

Use this to debug rules. This does increase overhead during planning.

See Also:

Constant Field Values

TRACE_STATS_PATH

public static final java.lang.String TRACE_STATS_PATH

Enables the planner to write out planner statistics for each planner phase and rule.

See Also:

Constant Field Values

defaultProperties

protected java.util.Map<java.lang.Object,java.lang.Object> defaultProperties

Field properties

checkpointTapRootPath

protected java.lang.String checkpointTapRootPath

defaultAssertionLevel

protected AssertionLevel defaultAssertionLevel

Field assertionLevel

defaultDebugLevel

protected DebugLevel defaultDebugLevel

Field debugLevel

Constructor Detail

FlowPlanner

public FlowPlanner()

Method Detail

getDefaultProperties

public java.util.Map<java.lang.Object,java.lang.Object> getDefaultProperties()

getDefaultConfig

public abstract Config getDefaultConfig()

getPlannerInfo

public abstract PlannerInfo getPlannerInfo(java.lang.String name)

getPlatformInfo

public abstract PlatformInfo getPlatformInfo()

initialize

public void initialize(FlowConnector flowConnector,
                       java.util.Map<java.lang.Object,java.lang.Object> properties)

buildFlow

public F buildFlow(FlowDef flowDef,
                   RuleRegistrySet ruleRegistrySet)

createFlow

protected abstract F createFlow(FlowDef flowDef)

getFlowStepFactory

public abstract FlowStepFactory<Config> getFlowStepFactory()

getFlowNodeFactory

public FlowNodeFactory getFlowNodeFactory()

configRuleRegistryDefaults

public void configRuleRegistryDefaults(RuleRegistry ruleRegistry)

resolveTails

protected Pipe[] resolveTails(FlowDef flowDef,
                              F flow)

resolveAssemblyPlanners

protected Pipe[] resolveAssemblyPlanners(FlowDef flowDef,
                                         Flow flow,
                                         Pipe[] pipes)

verifyAssembly

protected void verifyAssembly(FlowDef flowDef,
                              Pipe[] tails)

verifyAllTaps

protected void verifyAllTaps(FlowDef flowDef)

createFlowElementGraph

protected FlowElementGraph createFlowElementGraph(FlowDef flowDef,
                                                  Pipe[] flowTails)

verifySourceNotSinks

protected void verifySourceNotSinks(java.util.Map<java.lang.String,Tap> sources,
                                    java.util.Map<java.lang.String,Tap> sinks)

verifyTaps

protected void verifyTaps(java.util.Map<java.lang.String,Tap> taps,
                          boolean areSources,
                          boolean mayNotBeEmpty)

Method verifyTaps ...

Parameters:

taps - of type Map

areSources - of type boolean

mayNotBeEmpty - of type boolean

verifyPipeAssemblyEndPoints

protected void verifyPipeAssemblyEndPoints(FlowDef flowDef,
                                           Pipe[] flowTails)

Method verifyEndPoints verifies

there aren't dupe names in heads or tails. all the sink and source tap names match up with tail and head pipes

verifyTraps

protected void verifyTraps(FlowDef flowDef,
                           Pipe[] flowTails)

verifyCheckpoints

protected void verifyCheckpoints(FlowDef flowDef,
                                 Pipe[] flowTails)

verifyResult

public java.lang.Exception verifyResult(RuleResult ruleResult)

If there are rules for a given ProcessLevel on the current platform there must be sub-graphs partitioned at that level.

verifyResultInternal

protected void verifyResultInternal(RuleResult ruleResult)

handleExceptionDuringPlanning

protected PlannerException handleExceptionDuringPlanning(FlowDef flowDef,
                                                         java.lang.Exception exception,
                                                         FlowElementGraph flowElementGraph)

makeTempTap

protected Tap makeTempTap(java.lang.String name)

getDebugLevel

protected DebugLevel getDebugLevel(FlowDef flowDef)

getAssertionLevel

protected AssertionLevel getAssertionLevel(FlowDef flowDef)

makeTempTap

protected abstract Tap makeTempTap(java.lang.String prefix,
                                   java.lang.String name)