Table of Contents

Driven User Guide

version 1.2

Counter Data and Other Metrics in Tables

Tables map a lot of detailed information to all areas of application performance on Hadoop clusters. Tables show some basic information, which are listed as Common metrics in the column chooser. For example, a table on an Application View page provides flow name and status in each row. There are also specific Time and Duration metrics.

Driven can provide deeper insights in the tables if you choose to display data from counters. Counters feed big data application execution metrics to the Driven Plugin from Hadoop, Cascading, and other frameworks. Standard counters are listed as Diagnostic metrics in the column chooser. In addition, if custom counters are programmed in the application, Driven also surfaces these counters in the column chooser as importable attributes to your tables. Custom counter attributes are grouped toward the bottom of the column chooser.

Raw counter data is usually about a very granular level of an application run. For example, a counter could exist to report on the number of user profiles that are running an application. But the counter might be reporting metrics from separate steps of application instances one-by-one. You receive accurate data about the number of user profiles running the application at step level, but you do not get a total number of user profiles running applications cumulatively. The problem with receiving fragmented data like this is that you have an incomplete picture of the application. In addition, Hadoop holds on to the counter data for a short period of time. Without a storage mechanism, people and systems do not have one-stop access to historical data. The best workaround is to check logs, which is unwieldy and time-consuming.

A value of Driven is that it aggregates all the data reported from a counter into an application level, while also reporting on the flow and step level if the user wants to stay on the granular level. Driven maintains the historical data and makes it easily, dynamically retrievable. Driven allows you to filter so that you can slice and dice what you see from the counter data. On a higher level, you can easily compare counter data from one application to another application.

Note
 For information about importing counters to tables in the slice performance view, see Tuning Your Application.

To select or remove metrics in the displayed table:

  1. Click the column-chooser icon. Counter_Chooser

  2. Select the columns that you want to appear in the table. The standard metrics appear are grouped first in the list. Scroll down to view custom counters that can be reported if any were developed.

  3. Click UPDATE.

The following sections provide a brief explanation of the column metrics:

Common

The common metrics vary depending on whether you are viewing a table listing applications or flows.

  • ID - The numeric identifier (ID) of the application.

  • Owner - The owner (by name) of the application.

  • Slice Rate - The aggregated count of slices for an app in Running status and the amount of time that was measured. The slice rate is depicted by the vertical fluctuations of the line, which reflect changes during the application runtime. The count is displayed to indicate how many slices are running at a certain time.

  • Team - The team name that is associated with the application. The Owner of the team or appointed Team Leader is able to manage the team member details through a link in this column.

  • Type - The type of application.

  • Timeline - A timeline representation of each row with color coding to reflect the status. Hover over the graphed bar to display a pop-up of the status information.

  • Statements - Expressions (if any) that are used in the execution of a flow.

  • Job Tracker URL - Hyperlink to the Hadoop JobTracker dashboard, which tracks the performance of mappers and reducers.

Time

  • Finished Time - This is the time when Hadoop completes execution on the Cascading objects.

  • Pending Time - At the application level, Cascading objects (flows) are entered into a queue for work in Cascading. In this time, a Cascading object is waiting to be processed because it has a dependency on another Cascading object that needs reconciliation before it can be processed.

  • Run Time - This is the actual start time when Cascading objects are executed in Hadoop.

  • Start Time - This is when Cascading performs work on the Cascading objects, which includes the Query Planner mapping pipe assembly to the underlying computation fabric and incorporating additional Cascading API functionality.

  • Submit Time - This is when the Cascading objects are submitted to Hadoop for execution.

Diagnostic Counters

The Diagnostics metric is the collection of data points processed in Hadoop for the Cascading application.

  • Bytes Read - The number of bytes read during the execution of the application.

  • Bytes Written - The number of bytes written during the execution of the application.

  • Task Retries - The number of retries that Hadoop executed for a task.

  • Tuples Read - The number of tuples read during the execution of the application.

  • Tuples Spilled - The number of tuples that were written to less-efficient temporary disk storage because of memory constraints.

  • Tuples Trapped - The number of tuples trapped (data placed into an output file because of error in processing) during the execution of the application.

  • Tuples Written - The number of tuples written during the execution of the application.

Duration

The Duration metric is the product of measured time during a particular state and the matrix of ensuing states when processing the application.

  • Duration - The time when the application is in the Start status subtracted from the time when the application is in the Finished status.

  • Total Duration - The time when the application is in the Pending status subtracted from the time when the application is in the Finished status.

  • Pending:Run Duration - The time when the application is in the Pending status subtracted from the time when the application is in the Run status.

  • Pending Duration - The total time that the application is in Pending status.

  • Pending:Submit Duration - The time when the application is in the Pending status subtracted from the time when the application is in the Submitted status.

  • Running Duration - The total time that the application is in the Running status.

  • Start:Finish Duration - The time when the application is in the Started status subtracted from the time when the application is in the Finished status.

  • Start:Run Duration - The time when the application is in the Start status subtracted from the time when the application is in the Run status.

  • Started Duration - The total time that the application is in the Started status.

  • Submitted:Finish Duration - The time when the application is in the Submitted status subtracted from the time when the application is in the Finished status.

  • Submitted Duration - The total time that the application is in the Submitted status.

Tip
 See the Cascading application states documentation for more information about semantics for the counters.

table-metrics

Figure 1: This example displays some metrics, including one counter, that are available to monitor your Cascading applications

Next

Understanding the Anatomy of Your Application

© 2014-2015 Concurrent, Inc. All rights reserved.