Getting Started
version 1.3.8Getting Started
The Driven Agent is a collection JVM level agent libraries that allows monitoring of Apache Hadoop applications like Apache Hive or native MapReduce jobs within Driven.
The Agent is currently available for the following frameworks:
And is compatible with the following scheduler:
The Driven Agent is a Java agent wrapper for the Driven Plugin, and is available in two different forms, one that bundles the Driven plugin, and one that does not. See the Driven Documentation on details on the Driven Plugin.
Using the bundled agent is the quickest and simplest way to get started if the Driven Plugin has not already been installed for existing Cascading applications. In that case, the un-bundled agent can be used, which allows for the plugin to be upgraded and configured independently of the agent installation.
Downloading the Java agent
Select the agent library that is applicable to your framework.
From a terminal/shell, execute one of the following commands.
# latest Hive agent bundle
> wget -i http://files.concurrentinc.com/driven-agent/1.3/latest-driven-agent-hive-bundle.txt
# latest MapReduce agent bundle
> wget -i http://files.concurrentinc.com/driven-agent/1.3/latest-driven-agent-mr-bundle.txt
Note
|
The -i switch downloads the latest-…-bundle.txt file, and then
downloads the link in the file, which is always the latest release for the
current version of Driven.
|
Install the Java agent
To install the Agent:
-
Download the correct Driven Agent for your application type.
-
In your home directory, create a new directory named
driven-agent
. -
Copy the downloaded file into the
driven-agent
directory. -
Execute your app with the
-javaagent:/path/to/driven-agent-[framework]-<version>.jar
argument passed to the JVM. See framework specific instructions below. -
Log in to the Driven Server to see your application’s performance information.
Note, the Agent has to be enabled via a JVM level command line switch of the form below:
java -javaagent:/path/to/driven-agent-[framework]-<version>.jar[=key1=value1;key2=value2,value3] <other java arguments>
Where [framework] denotes Hive (hive) or MapReduce (mr), and <version> is the current agent version.
Putting the agent on the runtime CLASSPATH will have no effect.
Note
|
To work, place the -javaagent:/path/to/driven-agent-[framework]-<version>.jar switch on the JVM command
line before the application jar.
|
Configuring the Java agent
The Driven Agent accepts a number for configuration options after the path to the Driven Agent jar file.
java -javaagent:/path/to/driven-agent-[framework]-<version>.jar[=key1=value1;key2=value2,value3] <other java arguments>
Available Agent options can be printed to the console by running the Java Agent Jar with the following command:
java -jar /path/to/driven-agent-[framework]-<version>.jar
The Agent also accepts a properties file via the optionsFile
option. To generate a template file with defaults,
run the following command (with a dash as the only argument):
java -jar /path/to/driven-agent-[framework]-<version>.jar - > driven-agent.properties
Note
|
This re-directs the output to the file driven-agent.properties in the current directory.
|
Tip
|
The file specified by optionsFile will be treated relative to the Java JVM current working directory, and
if not found, will be relative to the Java agent directory, unless the path is absolute.
|
The configuration options break down into three groups.
Note
|
Some options may not be available for different frameworks. |
Agent Specific Options
optionsFile
-
file to read options values from, all values take precedence. relative to current dir, else agent jar directory
agentDisableLogging
-
disable all logging in agent and plugin
agentDebugLogging
-
enable debug logging in agent and plugin
agentExitOnlyOnJobCompletion
-
force jvm to remain active until the monitored jobs complete, fail, or are killed. appCompletionTimeout is not honored
agentKillJobsOnExit
-
on jvm exit, kill all running jobs. work will be marked as stopped if System.exit is called when detaching the client
agentLogSystemExitCalls
-
enable logging the stacktrace making System and Runtime exit calls. this installs a custom SecurityManager if none installed
Plugin Specific Options
drivenHosts
-
the server host names and ports to send data to: host1:80,host2:8080
drivenAPIKey
-
the API key to associate application executions with
drivenArchiveDir
-
the local directory to store copies of transmitted data
drivenDisabled
-
disable sending data to the server
drivenSuppressSliceData
-
disable sending slice level data and detailed low level performance visualizations, overrides server settings. this may reduce network traffic, load on any history servers, and indexing latency
drivenContinuousSliceData
-
enable frequent updates of slice level data before slice completion, update on completion is the default, overrides server settings. this may increase network traffic, load on any history server, and indexing latency
drivenSuppressJavaCommandData
-
disable sending the command line argument data, overrides server settings. this prevents sending sensitive information that may appear on the command line
Application Specific Options
appName
-
then name of this application. default is the jar name without version information
appVersion
-
then version of this application. default is parsed from the jar name
appTags
-
tags that should be associated with this application: cluster:prod,dept:engineering
appCompletionTimeout
-
timeout in millis to wait to send all completed application details
appFailedOnAnyUoWFail
-
if any unit-of-work fails, mark the application as failed. default is if the last unit-of-work fails the app is marked as failed
appFailedOnAnyUoWPending
-
if any unit-of-work has not been started, mark the application as failed. default is if the last unit-of-work fails the app is marked as failed