cascading.flow.planner
Class FlowPlanner<F extends Flow,Config>

java.lang.Object
  cascading.flow.planner.FlowPlanner<F,Config>

public abstract class FlowPlanner<F extends Flow,Config>
extends Object

Class FlowPlanner is the base class for all planner implementations.

Field Summary

protected AssertionLevel	assertionLevel Field assertionLevel
protected String	checkpointRootPath
protected DebugLevel	debugLevel Field debugLevel
protected Map<Object,Object>	properties Field properties

Constructor Summary

FlowPlanner()

Method Summary

abstract F	buildFlow(FlowDef flowDef) Method buildFlow renders the actual Flow instance.
protected ElementGraph	createElementGraph(FlowDef flowDef, Pipe[] flowTails)
protected abstract Flow	createFlow(FlowDef flowDef)
protected void	failOnGroupEverySplit(ElementGraph elementGraph)
protected void	failOnLoneGroupAssertion(ElementGraph elementGraph) Verifies that there are not only GroupAssertions following any given Group instance.
protected void	failOnMissingGroup(ElementGraph elementGraph)
protected void	failOnMisusedBuffer(ElementGraph elementGraph)
abstract Config	getConfig()
abstract PlatformInfo	getPlatformInfo()
Map<Object,Object>	getProperties()
protected PlannerException	handleExceptionDuringPlanning(Exception exception, ElementGraph elementGraph)
protected void	handleJobPartitioning(ElementGraph elementGraph) Inserts a temporary Tap between logical MR jobs.
protected void	handleJoins(ElementGraph elementGraph) Prevent leftmost sources from sourcing a downstream join on the rightmost side intra-task by inserting a temp tap between the left-sourced join and right-sourced join.
protected void	handleNonSafeOperations(ElementGraph elementGraph)
void	initialize(FlowConnector flowConnector, Map<Object,Object> properties)
protected void	insertTempTapAfter(ElementGraph graph, Pipe pipe) Method insertTapAfter ...
protected Tap	makeTempTap(String name)
protected abstract Tap	makeTempTap(String prefix, String name)
protected Pipe[]	resolveAssemblyPlanners(FlowDef flowDef, Flow flow, Pipe[] pipes)
protected Pipe[]	resolveTails(FlowDef flowDef, Flow<Config> flow)
protected void	verifyAllTaps(FlowDef flowDef)
protected void	verifyAssembly(FlowDef flowDef, Pipe[] tails)
protected void	verifyCheckpoints(FlowDef flowDef, Pipe[] flowTails)
protected void	verifyPipeAssemblyEndPoints(FlowDef flowDef, Pipe[] flowTails) Method verifyEndPoints verifies there aren't dupe names in heads or tails.
protected void	verifySourceNotSinks(Map<String,Tap> sources, Map<String,Tap> sinks)
protected void	verifyTaps(Map<String,Tap> taps, boolean areSources, boolean mayNotBeEmpty) Method verifyTaps ...
protected void	verifyTraps(FlowDef flowDef, Pipe[] flowTails)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

properties

protected Map<Object,Object> properties

Field properties

checkpointRootPath

protected String checkpointRootPath

assertionLevel

protected AssertionLevel assertionLevel

Field assertionLevel

debugLevel

protected DebugLevel debugLevel

Field debugLevel

Constructor Detail

FlowPlanner

public FlowPlanner()

Method Detail

getProperties

public Map<Object,Object> getProperties()

getConfig

public abstract Config getConfig()

getPlatformInfo

public abstract PlatformInfo getPlatformInfo()

initialize

public void initialize(FlowConnector flowConnector,
                       Map<Object,Object> properties)

createFlow

protected abstract Flow createFlow(FlowDef flowDef)

buildFlow

public abstract F buildFlow(FlowDef flowDef)

Method buildFlow renders the actual Flow instance.

Parameters:

flowDef -

Returns:

Flow

resolveTails

protected Pipe[] resolveTails(FlowDef flowDef,
                              Flow<Config> flow)

resolveAssemblyPlanners

protected Pipe[] resolveAssemblyPlanners(FlowDef flowDef,
                                         Flow flow,
                                         Pipe[] pipes)

verifyAssembly

protected void verifyAssembly(FlowDef flowDef,
                              Pipe[] tails)

verifyAllTaps

protected void verifyAllTaps(FlowDef flowDef)

createElementGraph

protected ElementGraph createElementGraph(FlowDef flowDef,
                                          Pipe[] flowTails)

verifySourceNotSinks

protected void verifySourceNotSinks(Map<String,Tap> sources,
                                    Map<String,Tap> sinks)

verifyTaps

protected void verifyTaps(Map<String,Tap> taps,
                          boolean areSources,
                          boolean mayNotBeEmpty)

Method verifyTaps ...

Parameters:

taps - of type Map

areSources - of type boolean

mayNotBeEmpty - of type boolean

verifyPipeAssemblyEndPoints

protected void verifyPipeAssemblyEndPoints(FlowDef flowDef,
                                           Pipe[] flowTails)

Method verifyEndPoints verifies

there aren't dupe names in heads or tails. all the sink and source tap names match up with tail and head pipes

verifyTraps

protected void verifyTraps(FlowDef flowDef,
                           Pipe[] flowTails)

verifyCheckpoints

protected void verifyCheckpoints(FlowDef flowDef,
                                 Pipe[] flowTails)

failOnLoneGroupAssertion

protected void failOnLoneGroupAssertion(ElementGraph elementGraph)

Verifies that there are not only GroupAssertions following any given Group instance. This will adversely affect the stream entering any subsequent Tap of Each instances.

failOnMissingGroup

protected void failOnMissingGroup(ElementGraph elementGraph)

failOnMisusedBuffer

protected void failOnMisusedBuffer(ElementGraph elementGraph)

failOnGroupEverySplit

protected void failOnGroupEverySplit(ElementGraph elementGraph)

handleExceptionDuringPlanning

protected PlannerException handleExceptionDuringPlanning(Exception exception,
                                                         ElementGraph elementGraph)

handleNonSafeOperations

protected void handleNonSafeOperations(ElementGraph elementGraph)

insertTempTapAfter

protected void insertTempTapAfter(ElementGraph graph,
                                  Pipe pipe)

Method insertTapAfter ...

Parameters:

graph - of type PipeGraph

pipe - of type Pipe

makeTempTap

protected Tap makeTempTap(String name)

makeTempTap

protected abstract Tap makeTempTap(String prefix,
                                   String name)

handleJobPartitioning

protected void handleJobPartitioning(ElementGraph elementGraph)

Inserts a temporary Tap between logical MR jobs.

Since all joins are at groups or splices, depth first search is safe

todo: refactor so that rules are applied to path segments bounded by taps todo: this would allow balancing of operations within paths instead of pushing todo: all operations up. may allow for consolidation of rules

Parameters:

elementGraph - of type PipeGraph

handleJoins

protected void handleJoins(ElementGraph elementGraph)

Prevent leftmost sources from sourcing a downstream join on the rightmost side intra-task by inserting a temp tap between the left-sourced join and right-sourced join.

Parameters:

elementGraph -