Driven User Guide

When you execute your Cascading application, the underlying framework builds a state model to optimally execute the flow on the Hadoop cluster. The Driven Plugin transmits the state model to the Driven application, which represents the Cascading execution plan as a DAG.

The DAG representation of each application execution can be useful to stakeholders responsible for documenting how an application has been developed and has performed, such as a documentation analyst. Over a period of time, the analyst might not be able to track and record relevant application details. Because Driven has a persistence layer to store application execution data, past application performance can be recreated by generating a DAG on demand. Without such an interface, it can be difficult to map business needs to technical implementation, especially in work environments that involve large teams that are spread across different regions.

In the graph, each node corresponds to a step or a processing function in your application code. You can refer to the specific code for a step by clicking on the node link.

The DAG on the app details page can be viewed in three different ways:

- The Contracted View is useful for complex and large applications.

- The Logical View (default) shows all the steps and taps (excluding implicit taps) and built-in functions.

- The Physical View shows all the steps including the implicit taps and built-in functions. The Physical View may show more details than the Logical View, if any exist.

Driven represents Cascading’s pipes metaphor as lines connecting steps in the DAG. A step is dependent on another step only if it relies on the execution of the previous step. Cascading dynamically determines the dependencies between the flows. If the output (sink) of one flow is consumed by another flow (as a source), Driven notates that dependency by connecting the two flows.

Visualizing your end-to-end application as a DAG along with operational data, such as read/write data processed at each step, can provide important insights into improving the performance of the application. For example, reviewing the DAG can expose opportunities to introduce Filter functions in your code upstream to reduce the volumes of the data being processed by the pipes or to make the Join functions more efficient.

Driven can refresh the displayed information as updates stream in from the plugin. This includes display of real-time progress of your application, which includes highlighting the current steps being executed, number of completed steps, and read/write data.

Getting the most current information can be very useful. You might discover in Driven that a long-running job is not executing properly, which could be a signal to terminate the application. Also, for example, if you see sudden slow-down in the progress of your application, you may want to immediately start investigating the reason (network storm or a rogue job submitted to the cluster).

One of the most interesting insights is the ability to track the percentage of applications that have completed in real-time. For long-running applications, it is often useful to spot-check the behavior to ensure that there are no anomalies.

Ensure that the Refresh toggle in the top right corner is enabled to allow the displayed Driven data to auto-refresh in real time. Click and slide the Refresh icon to toggle between enabled and disabled.

The table under the DAG provides a detailed breakdown of each flow in the application run. Some key monitoring assets of the tabular interface include the following capabilities: