cascading.flow.planner
Class FlowPlanner<F extends Flow,Config>

java.lang.Object
  extended by cascading.flow.planner.FlowPlanner<F,Config>

public abstract class FlowPlanner<F extends Flow,Config>
extends Object

Class FlowPlanner is the base class for all planner implementations.


Field Summary
protected  AssertionLevel assertionLevel
          Field assertionLevel
protected  String checkpointRootPath
           
protected  DebugLevel debugLevel
          Field debugLevel
protected  Map<Object,Object> properties
          Field properties
 
Constructor Summary
FlowPlanner()
           
 
Method Summary
abstract  F buildFlow(FlowDef flowDef)
          Method buildFlow renders the actual Flow instance.
protected  ElementGraph createElementGraph(FlowDef flowDef, Pipe[] flowTails)
           
protected abstract  Flow createFlow(FlowDef flowDef)
           
protected  void failOnGroupEverySplit(ElementGraph elementGraph)
           
protected  void failOnLoneGroupAssertion(ElementGraph elementGraph)
          Verifies that there are not only GroupAssertions following any given Group instance.
protected  void failOnMissingGroup(ElementGraph elementGraph)
           
protected  void failOnMisusedBuffer(ElementGraph elementGraph)
           
abstract  Config getConfig()
           
abstract  PlatformInfo getPlatformInfo()
           
 Map<Object,Object> getProperties()
           
protected  PlannerException handleExceptionDuringPlanning(Exception exception, ElementGraph elementGraph)
           
protected  void handleJobPartitioning(ElementGraph elementGraph)
          Inserts a temporary Tap between logical MR jobs.
protected  void handleJoins(ElementGraph elementGraph)
          Prevent leftmost sources from sourcing a downstream join on the rightmost side intra-task by inserting a temp tap between the left-sourced join and right-sourced join.
protected  void handleNonSafeOperations(ElementGraph elementGraph)
           
 void initialize(FlowConnector flowConnector, Map<Object,Object> properties)
           
protected  void insertTempTapAfter(ElementGraph graph, Pipe pipe)
          Method insertTapAfter ...
protected  Tap makeTempTap(String name)
           
protected abstract  Tap makeTempTap(String prefix, String name)
           
protected  Pipe[] resolveAssemblyPlanners(FlowDef flowDef, Flow flow, Pipe[] pipes)
           
protected  Pipe[] resolveTails(FlowDef flowDef, Flow<Config> flow)
           
protected  void verifyAllTaps(FlowDef flowDef)
           
protected  void verifyAssembly(FlowDef flowDef, Pipe[] tails)
           
protected  void verifyCheckpoints(FlowDef flowDef, Pipe[] flowTails)
           
protected  void verifyPipeAssemblyEndPoints(FlowDef flowDef, Pipe[] flowTails)
          Method verifyEndPoints verifies

there aren't dupe names in heads or tails.

protected  void verifySourceNotSinks(Map<String,Tap> sources, Map<String,Tap> sinks)
           
protected  void verifyTaps(Map<String,Tap> taps, boolean areSources, boolean mayNotBeEmpty)
          Method verifyTaps ...
protected  void verifyTraps(FlowDef flowDef, Pipe[] flowTails)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

properties

protected Map<Object,Object> properties
Field properties


checkpointRootPath

protected String checkpointRootPath

assertionLevel

protected AssertionLevel assertionLevel
Field assertionLevel


debugLevel

protected DebugLevel debugLevel
Field debugLevel

Constructor Detail

FlowPlanner

public FlowPlanner()
Method Detail

getProperties

public Map<Object,Object> getProperties()

getConfig

public abstract Config getConfig()

getPlatformInfo

public abstract PlatformInfo getPlatformInfo()

initialize

public void initialize(FlowConnector flowConnector,
                       Map<Object,Object> properties)

createFlow

protected abstract Flow createFlow(FlowDef flowDef)

buildFlow

public abstract F buildFlow(FlowDef flowDef)
Method buildFlow renders the actual Flow instance.

Parameters:
flowDef -
Returns:
Flow

resolveTails

protected Pipe[] resolveTails(FlowDef flowDef,
                              Flow<Config> flow)

resolveAssemblyPlanners

protected Pipe[] resolveAssemblyPlanners(FlowDef flowDef,
                                         Flow flow,
                                         Pipe[] pipes)

verifyAssembly

protected void verifyAssembly(FlowDef flowDef,
                              Pipe[] tails)

verifyAllTaps

protected void verifyAllTaps(FlowDef flowDef)

createElementGraph

protected ElementGraph createElementGraph(FlowDef flowDef,
                                          Pipe[] flowTails)

verifySourceNotSinks

protected void verifySourceNotSinks(Map<String,Tap> sources,
                                    Map<String,Tap> sinks)

verifyTaps

protected void verifyTaps(Map<String,Tap> taps,
                          boolean areSources,
                          boolean mayNotBeEmpty)
Method verifyTaps ...

Parameters:
taps - of type Map
areSources - of type boolean
mayNotBeEmpty - of type boolean

verifyPipeAssemblyEndPoints

protected void verifyPipeAssemblyEndPoints(FlowDef flowDef,
                                           Pipe[] flowTails)
Method verifyEndPoints verifies

there aren't dupe names in heads or tails. all the sink and source tap names match up with tail and head pipes


verifyTraps

protected void verifyTraps(FlowDef flowDef,
                           Pipe[] flowTails)

verifyCheckpoints

protected void verifyCheckpoints(FlowDef flowDef,
                                 Pipe[] flowTails)

failOnLoneGroupAssertion

protected void failOnLoneGroupAssertion(ElementGraph elementGraph)
Verifies that there are not only GroupAssertions following any given Group instance. This will adversely affect the stream entering any subsequent Tap of Each instances.


failOnMissingGroup

protected void failOnMissingGroup(ElementGraph elementGraph)

failOnMisusedBuffer

protected void failOnMisusedBuffer(ElementGraph elementGraph)

failOnGroupEverySplit

protected void failOnGroupEverySplit(ElementGraph elementGraph)

handleExceptionDuringPlanning

protected PlannerException handleExceptionDuringPlanning(Exception exception,
                                                         ElementGraph elementGraph)

handleNonSafeOperations

protected void handleNonSafeOperations(ElementGraph elementGraph)

insertTempTapAfter

protected void insertTempTapAfter(ElementGraph graph,
                                  Pipe pipe)
Method insertTapAfter ...

Parameters:
graph - of type PipeGraph
pipe - of type Pipe

makeTempTap

protected Tap makeTempTap(String name)

makeTempTap

protected abstract Tap makeTempTap(String prefix,
                                   String name)

handleJobPartitioning

protected void handleJobPartitioning(ElementGraph elementGraph)
Inserts a temporary Tap between logical MR jobs.

Since all joins are at groups or splices, depth first search is safe

todo: refactor so that rules are applied to path segments bounded by taps todo: this would allow balancing of operations within paths instead of pushing todo: all operations up. may allow for consolidation of rules

Parameters:
elementGraph - of type PipeGraph

handleJoins

protected void handleJoins(ElementGraph elementGraph)
Prevent leftmost sources from sourcing a downstream join on the rightmost side intra-task by inserting a temp tap between the left-sourced join and right-sourced join.

Parameters:
elementGraph -


Copyright © 2007-2015 Concurrent, Inc. All Rights Reserved.