Driven User Guide

When you execute your Cascading application, the underlying framework builds a rich state model to optimally execute the flow on the Hadoop cluster. The Driven plugin transmits the state model to the Driven application, where users can visualize their application as a Directed Acyclic Graph (DAG). Note that the DAG is automatically created by the Cascading layer and that the application developer does not author the application DAG shown by Driven through a separate interface.

In the graph, each node corresponds to a step or a processing function in your application code. You can refer to a specific code for a step by clicking on the node link.

Figure 2: Click the desired flow in the DAG to see details of the tap including the code line number.

Your application can be viewed in three different ways:

- The Contracted View is useful for complex and large applications.

- The Logical View (default) shows all the steps and taps (excluding implicit taps) and built-in functions.

- The Physical View shows all the steps including the implicit taps and built-in functions. The Physical View may show more details than the Logical View, if any exist.

Driven models Cascading’s pipes metaphor by connecting the steps with lines. Note that these dependencies are inferred by Cascading — a step is dependent on another step only if it relies on the execution of the previous step to process the data. Cascading dynamically determines the dependencies between the flows — if the output (sink) of one flow is consumed by another flow (as a source), Driven will notate that dependency by connecting the two flows.

Visualizing your end-to-end application as a DAG along with operational data such as data read and then written at each step can provide important insights into improving the performance of the application. For example, reviewing the DAG can expose opportunities to introduce Filter functions in your code upstream to reduce the volumes of the data being processed by the pipes or to make the Join functions more efficient.

Driven renders your application upon the start of your Cascading application. Driven provides real-time progress on your application, which includes highlighting the current steps being executed, number of steps completed, data read and written, among other things.

Get the most current information can be very useful — you can terminate a long-running job if you feel that it is not executing properly. Also, for example, if you see sudden slow-down in the progress of your application, you may want to immediately start investigating the reason (network storm or a rogue job submitted to the cluster).

Figure 3: This example shows the counter for the application being updated in real-time progress.

One of the most interesting insight is the ability to track the percentage of applications that have completed in real-time. For long-running applications (which are also very expensive), it is often useful to spot check the behavior to ensure that there are no anamolies.

Cascading-based applications benefit from the framework that creates a state model and collecting a rich set of instrumentation counters. Driven helps you visualize these counters in the right context to provide you with a methodology to tune your applications.

Figure 7: The Timeline view helps you quickly scan the flows of your application to uncover any bottlenecks.

The Timeline view provides a detailed dashboard to flows that comprise the application, helping you to quickly identify which part of your application needs attention (assuming you will first attempt to tune the more expensive parts of the application).

To help you understand if there are any processing latency in your application due to data, network, compute resources, or application logic, refer to the timing counters.