Table of Contents

Driven User Guide

version 1.2

Understanding the Anatomy of Your Application

The application details view shows all the flows and steps that are part of your Cascading application. (You can confirm that you are in the application details view if flows are listed in the table rows.) Driven aggregates the performance of individual mappers and reducers and frames them as metrics in the context of the overall application execution. This insight can facilitate easier application optimization and monitoring on the Hadoop cluster.

Application instances are represented as directed acyclic graphs (DAGs). The graph renders an interactive diagram of the flows and steps that can reveal underlying slice performance issues. Flow, step, and slice information are particularly useful to monitor instances of application execution over a period of time as the application grows in complexity and size. In addition, flow and step details on the DAG and in the table below the graph can be used to:

  • Understand real-time dependencies between steps and flows

  • Visualize your application, tracking steps in the graph to line numbers in your code

  • Investigate log error messages and stack exceptions

  • Tune application logic

Understanding the Graph

When you execute your Cascading application, the underlying framework builds a state model to optimally execute the flow on the Hadoop cluster. The Driven Plugin transmits the state model to the Driven application, which represents the Cascading execution plan as a DAG.


Figure 1: Sample directed acyclic graph of a Cascading application

The DAG representation of each application execution can be useful to stakeholders responsible for documenting how an application has been developed and has performed, such as a documentation analyst. Over a period of time, the analyst might not be able to track and record relevant application details. Because Driven has a persistence layer to store application execution data, past application performance can be recreated by generating a DAG on demand. Without such an interface, it can be difficult to map business needs to technical implementation, especially in work environments that involve large teams that are spread across different regions.

In the graph, each node corresponds to a step or a processing function in your application code. You can refer to the specific code for a step by clicking on the node link.


Figure 2: Click on a tap or other node in a flow of the DAG to see details

Viewing the Graph

Your application can be viewed in three different ways:

Contracted View - The Contracted View is useful for complex and large applications.

Logical View - The Logical View (default) shows all the steps and taps (excluding implicit taps) and built-in functions.

Physical View - The Physical View shows all the steps including the implicit taps and built-in functions. The Physical View may show more details than the Logical View, if any exist.

Driven represents Cascading’s pipes metaphor as lines connecting steps in the DAG. A step is dependent on another step only if it relies on the execution of the previous step. Cascading dynamically determines the dependencies between the flows. If the output (sink) of one flow is consumed by another flow (as a source), Driven notates that dependency by connecting the two flows.

Visualizing your end-to-end application as a DAG along with operational data, such as read/write data processed at each step, can provide important insights into improving the performance of the application. For example, reviewing the DAG can expose opportunities to introduce Filter functions in your code upstream to reduce the volumes of the data being processed by the pipes or to make the Join functions more efficient.

Real-Time Visibility into Your Application

Driven starts rendering data about your Cascading application as soon as you start it. Driven displays real-time progress of your application, which includes highlighting the current steps being executed, number of completed steps, and read/write data.

If you want the Driven view to automatically update for an application in active state, ensure that this capability is enabled. Hover over and click the refresh icon refresh_icon_DAG by the DAG to toggle between enabled and disabled.

Getting the most current information can be very useful. You might discover in Driven that a long-running job is not executing properly, which could be a signal to terminate the application. Also, for example, if you see sudden slow-down in the progress of your application, you may want to immediately start investigating the reason (network storm or a rogue job submitted to the cluster).

One of the most interesting insights is the ability to track the percentage of applications that have completed in real-time. For long-running applications, it is often useful to spot-check the behavior to ensure that there are no anomalies.

Status State of the Application

As the application runs, it transitions through various states. The status state is instantly displayed in Driven. The status state includes:

Pending_Started_State - The icon for the Started status and the Pending status is the same. Started status indicates that Cascading has started work on the application. Pending status indicates that the application is queued for processing.

Submitted_State - The Submitted status indicates that the application has been submitted to Hadoop for execution.

Running_State - The Running status indicates that Hadoop is executing the application.

Successful_State - The Successful status indicates that the application ran successfully.

Stopped_State - The Stopped status indicates that the application was stopped.

Failed_State - The Failed status indicates that the application failed to be executed by Hadoop.

Stack Trace

For applications in FAILED status, you can view the stack trace of these applications to further investigate for errors. Click the Show stack trace error icon stack-trace-info in the upper right corner to display the stack trace error information.

Flow Table

The table under the DAG provides a detailed breakdown of each flow in the application run. Some key monitoring assets of the tabular interface include the following capabilities:

  • Click on a hyperlinked flow name to focus on component slice performance, JobTrackers, and node statistics

  • Segment flow data with correlated JobTracker, step, and customized counter details

Uncovering Bottlenecks with the Timeline View

Driven helps you visualize instrumentation counters in a context to help you tune your applications.


Figure 3: The Timeline view helps you quickly scan the flows of your application to uncover any bottlenecks

The Timeline view provides a detailed dashboard of flows that comprise the application, helping you to quickly identify which part of your application needs attention (assuming you will first attempt to tune the more resource-draining parts of the application).

The Driven page displays a maximum of 25 rows. Use the pagination arrows to navigate a table that spans more than one page.


The columns can be arranged by drag and drop to your preferred order after clicking on a column heading. You can also sort the items within a column by clicking on the bidirectional arrow.

Importing Counter Data and Other Metrics

Driven lets you customize most of the information that the table displays. Click the column chooser icon Counter_Chooser to reveal or conceal columnar metrics. The Status and Name columns cannot be hidden.

The columnar metrics are categorized in the column chooser. Each category can be collapsed or expanded.

See Counter Data and Other Metrics in Tables for more information.