Table of Contents

Installation guide for Driven plugin

version 1.0.5

Overview

Driven is a web-application that helps you visualize the operational details around all phases of your Cascading application: application development, debugging, performance tuning, operator monitoring. The Driven application receives telemetry data from your running Cascading applications (through the Driven plugin) and analyzes the signals to provide you meaningful insights.

You do not need to make any changes to your existing Cascading applications to integrate with the Driven application. To use Driven, the Driven Plugin must be visible to your Cascading application. Driven is free for development use.

Depending on how your Driven application is deployed, please follow the appropriate sequence of instructions:

Cascading application uses self-hosted version of Driven application

Cascading application uses cascading.io, Concurrent’s hosted service

Cascading application runs on Amazon Elastic MapReduce

Using self-hosted version of Driven

  1. To get the latest version of the Driven Plugin jar for self-hosted version (from Unix or Mac OS), invoke:

    > wget -i http://files.concurrentinc.com/driven/1.0/driven-plugin/latest-shaded-jar.txt

  2. If you have not done it already, log into the Driven application, and click on the user icon on the top right corner of the landing page. Select Team list, and note the key associated with your team. Please note that features associated with Teams are only available for the on-premise version of the Driven application.

If you are running your application on Hadoop

  1. Make API key visible to the Driven Plugin: Create cascading-service.properties in your Hadoop configuration directory (referred later as $HADOOP_CONF). For Hadoop 1.x, this may be $HADOOP_HOME/conf and for Hadoop 2.x, it will be $HADOOP_INSTALL/etc/hadoop

    > echo cascading.management.document.service.apikey=_YOUR_API_KEY_ >> $HADOOP_CONF/cascading-service.properties

  2. Make the plugin available by adding the following property to $HADOOP_CONF/cascading-service.properties

    > echo cascading.management.service.jar=path/to/driven-plugin-1.0.5-io.jar > $HADOOP_CONF/cascading-service.properties

    Alternatively, you can set the path to the plugin through the $HADOOP_CLASSPATH environment variable

    > export HADOOP_CLASSPATH=path/to/driven-plugin-1.0.5-io.jar

  3. Finally, set the host URL key.

    > echo cascading.management.document.service.hosts=_YOUR_DRIVEN_SERVER_URL_ >> $HADOOP_CONF/cascading-service.properties

If you are running your application in Local Mode

  1. Expose the Driven API key by either by setting a local environment variable:

    export DRIVEN_API_KEY=_YOUR_API_KEY_

    or by setting a Java JVM level system property named cascading.management.document.service.apikey:

    > java -Dcascading.management.document.service.apikey=_YOUR_API_KEY_ ....

  2. Next, add the Driven plugin to your classpath. The plugin only needs to be placed into the CLASSPATH environment variable or specified on the java command line using the -cp flag.

    To run, you will have a command line similar to this:

    > java -cp your.jar:driven-plugin-1.0.5-io.jar your.Main arg1 arg2

  3. Finally, specify how the plugin with reach the Driven application. This can be done either by setting a local environment variable:

    export DRIVEN_SERVER_HOSTS=_YOUR_DRIVEN_SERVER_URL_

    By setting a Java JVM level system property named cascading.management.document.service.hosts:

    > java -Dcascading.management.document.service.hosts=_YOUR_DRIVEN_SERVER_URL_ ....

If you wish to package the Driven Plugin directly into your application, you may simply add the Maven spec to your project file as a dependency. This is particularly useful if you plan to use Driven from your IDE as you develop your Cascading application. See Configuring Maven and Gradle for details.

NOTE: There are multiple ways to integrate your Driven plugin jar into your application, so it is always wise to
verify that you are picking up the file from the correct location. When your Cascading application runs, it will log
the path of the Driven Plugin jar file through the properties 'ServiceLoader', 'CascadingService', and 'DrivenDocumentService'.
Ensure that all three properties refer to the same driven plugin.

Now, when you run your Cascading application, you will be able to visualize its operational details by going to the URL for your Driven application.

Using cascading.io service

If you will be using Driven through Concurrent’s hosted service, cascading.io, your Cascading application must have network access to the Internet so it may send data to the domain name "driven.cascading.io" over port 443 (the default SSL port for HTTP). Note that the client side of Cascading will be the only Java process attempting to make a remote network connection. Neither Cascading or the Plugin will open an connection from within an Apache Hadoop cluster.

The simplest way to get started is by using Driven on cascading.io without an account. Once working, create an account and get an API Key for use with the Plugin to enable more features such as comparing performance of an application over historical runs.

  1. To get the latest version of the Driven Plugin jar for cascading.io (from Unix or Mac OS), invoke:

    > wget -i http://files.concurrentinc.com/driven/1.0/driven-plugin/latest-jar.txt

    This downloads the Driven Plugin fat jar, which includes all dependencies.

If you are running your application on Hadoop

  1. Make the plugin available by adding the following property to $HADOOP_CONF/cascading-service.properties

    > echo cascading.management.service.jar=path/to/driven-plugin-1.0.5-io.jar > $HADOOP_CONF/cascading-service.properties

    Alternatively, you can set the path to the plugin through the HADOOP_CLASSPATH environment variable

    > export HADOOP_CLASSPATH=path/to/driven-plugin-1.0.5-io.jar

If you are running your application in Local Mode

  1. Add the Driven plugin to your classpath. If you have an existing Cascading local-mode application, the plugin only needs to be placed into the CLASSPATH environment variable or specified on the java command line using the -cp flag.

    To run, you will have a command line similar to this:

    > java -cp your.jar:driven-plugin-1.0.5-io.jar your.Main arg1 arg2
NOTE: There are multiple ways to integrate your Driven plugin jar into your application, so it is always wise to
verify that you are picking up the file from the correct location. When your Cascading application runs, it will log
the path of the Driven Plugin jar file through the properties 'ServiceLoader', 'CascadingService', and 'DrivenDocumentService'.
Ensure that all three properties refer to the same driven plugin.

Depending on your platform either local or Hadoop, you run your application as you would normally. When the your application starts, you’ll see a message in the console logs that gives you a link to view your application on the Driven server:

    ________        __    ________        .__
   /  _____/  _____/  |_  \______ \_______|__|__  __ ____   ____
  /   \  ____/ __ \   __\  |    |  \_  __ \  \  \/ // __ \ /    \
  \    \_\  \  ___/|  |    |    `   \  | \/  |\   /\  ___/|   |  \
   \______  /\_____>__|   /_______  /__|  |__| \_/  \_____>___|  /
          \/                      \/                           \/ ©

  Follow this link to view your application:

  https://driven.cascading.io/driven/[your app ID]

At any time, you can visit the URL to view your application in Driven. Note that every time Cascading is executed, your application will be given a random unique application ID (or app ID). This ID uniquely identifies your application instance. The next time the application is run, a new ID will be generated.

Also, the full banner will not be displayed if you have configured the API key by registering for a Driven account.

With a Driven account

By registering for a Driven account on Driven web site, you will be able to view information about your historical application runs. This can become very useful when looking for outlier behavior in your application performance.

  1. Create an account by registering at the Driven web site (Driven is free for development use)

  2. After creating an account and logging in to Driven, you will see your account name at the top right of the screen. Mousing over the account name will given the option to display your API Key.

If you are running your application on Hadoop

  1. Make API key visible to the Driven Plugin: Create cascading-service.properties in your Hadoop configuration directory (referred later as $HADOOP_CONF). For Hadoop 1.x, this may be $HADOOP_HOME/conf and for Hadoop 2.x, it will be $HADOOP_INSTALL/etc/hadoop

    > echo cascading.management.document.service.apikey=_YOUR_API_KEY_ >> $HADOOP_CONF/cascading-service.properties

If you are running your application in Local Mode

  1. Expose the Driven API key by either by setting a local environment variable:

    export DRIVEN_API_KEY=_YOUR_API_KEY_

    or by setting a Java JVM level system property named cascading.management.document.service.apikey:

    > java -Dcascading.management.document.service.apikey=_YOUR_API_KEY_ ....

Running on Amazon EMR

  1. You can use the Driven plugin with Amazon Elastic MapReduce by adding this to the elastic-mapreduce client command line arguments:

    --bootstrap-action s3://files.concurrentinc.com/driven/1.0/driven-plugin/install-driven-plugin.sh

    This installation method installs the plugin via the Hadoop configuration so that Cascading applications launched from the Elastic MapReduce web console or elastic-mapreduce command line client will pickup the plugin.

  2. Alternatively, if you pre-installed the plugin, you can pass the value an argument:

    --bootstrap-action s3://files.concurrentinc.com/driven/1.0/driven-plugin/install-driven-plugin.sh --args "--api-key,_YOUR_API_KEY_,--host,_YOUR_DRIVEN_SERVER_URL_"

Maven and Gradle Dependencies

The Driven Plugin is available through our Maven compatible repository.

Note that it is important to specify the proper artifact classifier. The Driven Plugin is provided as a self contained fat jar with all its dependencies embedded. See the build file notes for restrictions on fatJar, shading, uberjar and other means of repackaging the Driven plugin.

Maven

In the <repositories> block add:

    <repository>
      <id>conjars</id>
      <url>http://conjars.org/repo</url>
    </repository>

and then add the dependency:

<dependency>
  <groupId>driven</groupId>
  <artifactId>driven-plugin</artifactId>
  <version>1.0.5</version>
  <classifier>io</classifier>
</dependency>

Gradle

mavenRepo name: 'conjars', url: 'http://conjars.org/repo'

And then add the dependency:

compile group: 'driven', name: 'driven-plugin', version: '1.0.5', classifier:'io'

Getting Support

Support for Driven is provided via a public forum:

or by emailing:

FAQ

For topics not covered here, checkout the FAQ.

Driven FAQ