StatsD agent tutorial

This tutorial demonstrates how to use the Zenoss backend for gostatsd to send data to Zenoss Cloud. Along the way, you'll learn how to use metadata to unlock the power of Zenoss Cloud.

To perform the procedures in this tutorial, you need the following:

A login on a Linux host or virtual machine that has
- internet access
- Docker CE or Docker EE installed (this example uses Docker CE)
- (optional) a terminal multiplexer such as screen or tmux
A Zenoss Cloud account with the Manager role
An authentication key
Familiarity with Zenoss Cloud dashboards
(Optional) Familiarity with StatsD metrics
(Optional) Familiarity with Zenoss Cloud entities

Most users take about an hour to complete this tutorial. Let's get started!

Download required images and create a container network

This tutorial relies on the following images:

An Alpine image or any Linux image that includes the netcat utility
The Zenoss image of the Atlassian gostatsd server

Follow these steps:

Log in to a Linux system as a user that has sudo privileges or that is a member of the docker group.

If the user is a member of the docker group, you can omit sudo from all the command invocations in this tutorial.
Pull the Alpine image.
```
sudo docker pull alpine:latest
```

Pull the Zenoss image.

sudo docker pull zenoss/gostatsd:latest

Verify that the images are present locally.
```
sudo docker images
```
Create a Docker network for container communications, if necessary.

This tutorial uses a bridge network named mynet. You can use a different name if you prefer.
1. Determine whether the network already exists.
```
sudo docker network list | grep mynet
```
  If the command returns a result and the driver is Bridge, skip the next substep.
2. Create the network:
```
sudo docker network create --driver bridge mynet
```

Continue to the next section.

Review StatsD internal metrics

In this section, you'll start the server to verify it's working properly, and display the internal metrics that the server produces, to review their syntax.

Use the following command to start an interactive container for the gostatsd server. The gostatsd options configure it to send data points to STDOUT every 10 seconds:

sudo docker run --rm --name zenoss_gostatsd --net mynet \
  -ti zenoss/gostatsd:latest --backends=stdout --verbose --flush-interval=10s

Once you have some output, enter CTRL-c to stop the container.

Here's the syntax of the metrics:

| Severity [ Index ]: Bucket-Name Value Timestamp

Each unique bucket is a separate metric. Here are some of the bucket names from a sample run:

stats.gauge.statsd.aggregator.aggregation_time.aggregator_id.0.s.6a7d96c3674e
stats.gauge.statsd.aggregator.metricmaps_received.aggregator_id.0.s.6a7d96c3674e
stats.gauge.statsd.aggregator.metrics_received.aggregator_id.0.s.6a7d96c3674e
stats.gauge.statsd.aggregator.process_time.aggregator_id.0.s.6a7d96c3674e
stats.gauge.statsd.aggregator.reset_time.aggregator_id.0.s.6a7d96c3674e
stats.gauge.statsd.channel.avg.aggregator_id.0.channel.dispatch_aggregator.s.6a7d96c3674e
stats.gauge.statsd.channel.avg.channel.backend_events_sem.s.6a7d96c3674e
stats.gauge.statsd.channel.capacity.aggregator_id.0.channel.dispatch_aggregator.s.6a7d96c3674e
stats.gauge.statsd.channel.capacity.channel.backend_events_sem.s.6a7d96c3674e

The server produces quite a few metrics about itself, and when you start the server with the Zenoss backend, data points for these internal metrics are sent to Zenoss Cloud.

Continue to the next section.

Send internal metrics to Zenoss Cloud and display them in a dashboard

In this section, you'll configure the gostatsd server to send its internal metrics to Zenoss Cloud, and then display the metrics in a Zenoss Cloud dashboard.

In Zenoss Cloud, an entity can have one or more metrics. You'll define an entity for the gostatsd server and associate some key metadata with the entity. In Zenoss Cloud, you'll use the metadata to define the dashboard scope and then display some internal metrics.

Note

You need an authentication key to configure the server.

Create tags

One of the key advantages of the gostatsd implementation of StatsD is its tags feature. Tags are attribute-value pairs separated by the colon character (:). The Zenoss backend parses tags to create entity and metadata objects. In Zenoss Cloud, the tags that contribute to uniquely identifying an entity are dimension attributes. All other tags are metadata attributes.

In gostatsd, the name of a tag can be any string, which Zenoss Cloud supports. However, the policy service that processes streaming data in Zenoss Cloud reserves the following tag names:

Tag name	Description	Required?	Notes
`source-type`	The class name of the application sending data.	Yes	For `gostatsd`, use `com.atlassian.gostatsd`. The policy service uses this value to determine which policy to apply.
`source`	An instance identifier for source-type.	Yes	Must be unique in your environment. Source is a key field in Smart View and dashboard queries.
`name`	The display name for an entity.	No	Not required to be unique. Without this tag, entities are labeled `[ UNNAMED ]` in Smart View.

A required tag can be used either as a dimension attribute or as a metadata attribute.

Tip

Why bother with multiple dimension attributes, anyway? A globally-unique identifier would work and could be easier to implement. The answer is search—each dimension attribute becomes another name and value that can be used separately or in combination to find an entity. In fact, all entity and metric metadata can be used in queries.

Tags for dimension attributes

To begin, create tags for dimension attributes, so that the Zenoss Cloud entity for your gostatsd server will be unique to your organization.

The source tag is commonly used as a dimension attribute. Let's use something descriptive:

source:statsd.tutorial.server

That's useful but you need more to ensure that you have a unique entity. Add another tag with the local part of your company email address:

my-local-part:<MyLocalPart>

Good. These two tags uniquely identify your gostatsd server in your organization.

Tags for metadata attributes

Now, create tags for metadata attributes.

You have to include the source-type tag, and you know what value to use:

source-type:com.atlassian.gostatsd

Though not technically required, the name tag is very helpful. You can make name very simple by using your email again:

name:<MyLocalPart>.server

These two metadata attributes are enough for this tutorial.

Configure the server

Create a file for gostatsd configuration variables:

cat <<EOF > ${HOME}/myvars
GSD_VERBOSE=1
GSD_BACKENDS=stdout zenoss
GSD_FLUSH_INTERVAL=30s
GSD_MAX_WORKERS=1
EOF

The file follows the syntax defined by Docker.

Add the tags to the configuration file.

The GSD_INTERNAL_TAGS variable associates tags with internal metrics. Make sure you've replaced <MyLocalPart> with a real value:

echo "GSD_INTERNAL_TAGS=source:statsd.tutorial.server my-local-part:<MyLocalPart> source-type:com.atlassian.gostatsd name:<MyLocalPart>.server" >> ${HOME}/myvars

Specify which tags combine to uniquely identify the entity.

The rest of the variables you'll specify are specific to the Zenoss backend of gostatsd. The ZENOSS_METRIC_DIMENSION_TAGS variable is used to assign dimension attributes:
```
echo "ZENOSS_METRIC_DIMENSION_TAGS=source my-local-part" >> ${HOME}/myvars
```
Specify which tags provide metadata for the entity.
1. The tags assigned to ZENOSS_MODEL_DIMENSION_TAGS identify the entity, so they must match the tags assigned to ZENOSS_METRIC_DIMENSION_TAGS.
```
echo "ZENOSS_MODEL_DIMENSION_TAGS=source my-local-part" >> ${HOME}/myvars
```
2. The tags assigned to ZENOSS_MODEL_METADATA_TAGS specify the metadata attributes for the entity.
```
echo "ZENOSS_MODEL_METADATA_TAGS=source-type name" >> ${HOME}/myvars
```
Specify the Zenoss API endpoint variable.

Replace YOUR-API-ENDPOINT with the [Zenoss API endpoint]/api/zenoss-api.html#zenoss-api-endpoints) for your organization.
```
echo "ZENOSS_ADDRESS=YOUR-API-ENDPOINT:443" >> ${HOME}/myvars
```
Specify the authentication key variable.

Open ${HOME}/myvars with a text editor such as vim and paste the key into the file. Replace <MyAPIKey> with your authentication key:
```
ZENOSS_API_KEY=<MyAPIKey>
```

Review the variables file. Your file should look like this example:

GSD_VERBOSE=1
GSD_BACKENDS=stdout zenoss
GSD_FLUSH_INTERVAL=30s
GSD_MAX_WORKERS=1
GSD_INTERNAL_TAGS=source:statsd.tutorial.server my-local-part:<MyLocalPart> source-type:com.atlassian.gostatsd name:<MyLocalPart>.server
ZENOSS_METRIC_DIMENSION_TAGS=source my-local-part
ZENOSS_MODEL_DIMENSION_TAGS=source my-local-part
ZENOSS_MODEL_METADATA_TAGS=source-type name
ZENOSS_ADDRESS=YOUR-API-ENDPOINT:443
ZENOSS_API_KEY=<MyAPIKey>

Start the server.

The server sends data points to Zenoss Cloud and STDOUT. The grep filter in the following invocation gives you a way to track progress locally (the first output takes 60 seconds).
```
sudo docker run --rm --name zenoss_gostatsd --env-file ${HOME}/myvars \
  --net mynet -ti zenoss/gostatsd:latest | grep -E 'process_time|ERRO'
```

Display the internal metrics

OK! You're ready to create a Zenoss Cloud dashboard and view the server's internal metrics. (For more information about dashboards, see Dashboards and Dashboard tutorials.)

In Zenoss Cloud, click DASHBOARDS.
Click the panel toggle, and then click NEW DASHBOARD.
Define the dashboard scope.
1. In the upper-left corner, enter a descriptive name.
2. Click ADD ENTITIES.
3. In the Other sources field, enter statsd and then s elect the name of your source from the list that displays beneath the field.
4. Click the check mark to accept the source.
5. In the upper-right corner, click NEXT.
6. In the dashboard editor, click SAVE in the upper-right corner.
You need to set the time range and refresh rate before adding tiles.
In the dashboard header, click the time range button.

Note that the scope button shows that the current scope includes 1 entity.
Set the time range to 1H (1 hour), and then click the time range button again, to hide it.
Click the auto refresh icon.
Enable auto-refresh, and then set the frequency to 1 minute.
Click the dashboard editor icon, and then add tiles.
In the AVAILABLE TILES palette, click Graph.
In the TILE CONFIGURATION dialog, specify the metric to display.
1. In the Entity and metric field, enter the value of the name tag, <MyLocalPart>.server.
2. Select <MyLocalPart>.server from the list that displays beneath the field.
3. In the Entity and metric field, enter a metric name.
4. In the remaining fields, specify additional information as desired, and then click SAVE.
Add Graph tiles as desired.
In the upper-right corner of the dashboard, click SAVE.

Send sample application data to Zenoss Cloud and update the dashboard

You've succeeded in sending gostatsd internal metrics to Zenoss Cloud! That's no reason to use gostatsd; let's add a sample application. You'll simulate an application by sending data points for two metrics to your gostatsd server, and then add the metrics to your dashboard.

The following script (/tmp/sample-app.sh) is the initial version of our sample application, which sends just two StatsD metrics:

##!/bin/sh
RANDOM=$$
while :
do
  echo "random.count:${RANDOM}|c" | nc -u -w1 zenoss_gostatsd 8125
  echo "random.number:${RANDOM}|g" | nc -u -w1 zenoss_gostatsd 8125
  echo -n "."
  sleep 3
done

The script uses netcat to send data points to the gostatsd server in the zenoss_gostatsd container every 3 seconds.

You'll create a new entity in Zenoss Cloud for this sample application. It's not complicated, but it is instructive: You'll work through creating tags again, learn a new option for configuring the server, and learn about entity relationships.

Create tags

Once again, start by creating tags for the dimension attributes of your new entity.

Tags for dimension attributes

Use the source tag here, too, but with a different value. The following tag takes advantage of the tutorial namespace you used for the server:

source:statsd.tutorial.sample-app

Use the my-local-part tag again to make your entity unique:

my-local-part:<MyLocalPart>

Tags for metadata attributes

You have to include the source-type tag. Here, you'll extend the namespace by appending additional information:

source-type:com.atlassian.gostatsd.tutorial.sample-app

And of course, you want a display title for your entity, so you'll also specify the name tag:

name:<MyLocalPart>.sample-app

Here's the updated script with all the tags:

##!/bin/sh
RANDOM=$$
SOURCE_TYPE="source-type:com.atlassian.gostatsd.tutorial.sample-app"
SOURCE="source:statsd.tutorial.sample-app"
MY_LOCAL_PART="<MyLocalPart>"
NAME="name:<MyLocalPart>.sample-app"
while :
do
  echo "random.count:${RANDOM}|c|#${SOURCE_TYPE},${SOURCE},${MY_LOCAL_PART},${NAME}" | nc -u -w1 zenoss_gostatsd 8125
  echo "random.number:${RANDOM}|g|#${SOURCE_TYPE},${SOURCE},${MY_LOCAL_PART},${NAME}" | nc -u -w1 zenoss_gostatsd 8125
  echo -n "."
  sleep 3
done

Make sure you've replaced <MyLocalPart> with a real value.

Configure the server

Let's review the server configuration:

GSD_VERBOSE=1
GSD_BACKENDS=stdout zenoss
GSD_FLUSH_INTERVAL=30s
GSD_MAX_WORKERS=1
GSD_INTERNAL_TAGS=source-type:com.atlassian.gostatsd source:statsd.tutorial.server my-local-part:<MyLocalPart> name:<MyLocalPart>.server
ZENOSS_METRIC_DIMENSION_TAGS=source my-local-part
ZENOSS_MODEL_DIMENSION_TAGS=source my-local-part
ZENOSS_MODEL_METADATA_TAGS=source-type name
ZENOSS_ADDRESS=YOUR-API-ENDPOINT:443
ZENOSS_API_KEY=<MyAPIKey>

You're using tags for the sample application script the same way you're using them for the server, so nothing needs to change...well, make sure you've replaced <MyLocalPart> with a real value, of course.

Continue to the next section.

Simulate an application

In the terminal session where the gostatsd server is running, restart the server. Restarting the server allows it to pick up the new configuration.
1. Enter CTRL-c to stop the server.
2. Start the server again. This invocation includes a filter to display the sample application metrics:
```
sudo docker run --rm --name zenoss_gostatsd --env-file ${HOME}/myvars \
  --net mynet -ti zenoss/gostatsd:latest | grep -E 'sample-app|ERRO'
```
Create a second terminal session, either with a terminal multiplexer or by other means.

In the new session, start an interactive Alpine container.

sudo docker run --rm --name myalpine --net mynet -ti alpine:latest

In the Alpine container, create a script.

Use a text editor to create /tmp/sample-app.sh.

#!/bin/sh
RANDOM=$$
SOURCE_TYPE="source-type:com.atlassian.gostatsd.tutorial.sample-app"
SOURCE="source:statsd.tutorial.sample-app"
MY_LOCAL_PART="<MyLocalPart>"
NAME="name:<MyLocalPart>.sample-app"
while :
do
  echo "random.count:${RANDOM}|c|#${SOURCE_TYPE},${SOURCE},${MY_LOCAL_PART},${NAME}" | nc -u -w1 zenoss_gostatsd 8125
  echo "random.number:${RANDOM}|g|#${SOURCE_TYPE},${SOURCE},${MY_LOCAL_PART},${NAME}" | nc -u -w1 zenoss_gostatsd 8125
  echo -n "."
  sleep 3
done

Set execute permission on the script.
```
chmod +x /tmp/sample-app.sh
```
Start the script:
```
/tmp/sample-app.sh
```

Continue to the next section.

Update the dashboard

In Zenoss Cloud, click DASHBOARDS.
Open the dashboard you created in the previous section.
Verify that the time range is PAST 1 HOUR and that auto-refresh is enabled at 1 minute.
Update the dashboard scope.
1. In the Source field, add the source of the new entity.
  
  The source name is statsd.tutorial.sample-app.
2. In the upper-right corner, click SAVE.
Add a pair of Graph tiles to the dashboard.

The entity name to find is <MyLocalPart>.sample-app. Note that the random.count metric is split into two metrics, random.count.count and random.count.rate. This is characteristic of how the Zenoss backend handles metrics.

Add another Graph tile to the dashboard.

Use the following configuration:

Tile title:	My Random Count
Chart type:	line
Legend:	bottom
Metric name:	Enter `random.count.co`, select `random.count.count`
Aggregator:	none
Entity types:	(unchanged)
ADVANCED:	(unchanged)

Click SAVE.

In the dashboard header, click SAVE.

The new tile includes a link to the Smart View of the new entity, named <MyLocalPart>.sample-app.
Click the<MyLocalPart>.sample-app link.

In Smart View, there are no related entities! It's plain that the gostatsd server entity is related to the sample application entity, but not plain to Zenoss Cloud. In the next section, you'll add tags to the server configuration and the gauge script so that Zenoss Cloud knows the two are related.

Continue to the next section.

Create a dependency relationship between entities

The streaming data feature of Zenoss Cloud supports defining a dependency relationship between one entity and one or more other entities. In this section, you'll add tags to define a dependency relationship, and then restart the server and view the result.

Create tags

Zenoss Cloud provides the following tags for defining unidirectional dependency relationships.

Tag name	Description
simpleCustomRelationshipSourceTag	An entity that is a dependency for one or more other entities
simpleCustomRelationshipSinkTag	An entity that is dependent on another entity

The value of both tags can be anything at all but the values on each must be identical. Descriptive values are helpful for debugging; let's use the following string:

statsd.tutorial.<MyLocalPart>

Remember to replace <MyLocalPart> with a real value.

Strictly speaking, the sample application is dependent on the server, because without the server, the application metrics would not make it to Zenoss Cloud. Normally, the kind of dependency that you want to establish would be between, for example, an application server and a backend database. For this tutorial, you'll specify the server as the source entity and the sample application script as the sink entity.

Configure the server

First, add the relationship source tag to the server internal metrics:

GSD_INTERNAL_TAGS=source-type:com.atlassian.gostatsd source:statsd.tutorial.server my-local-part:<MyLocalPart> name:<MyLocalPart>.server simpleCustomRelationshipSourceTag:statsd.tutorial.<MyLocalPart>

You're adding metadata attributes, so you have to update the ZENOSS_MODEL_METADATA_TAGS variable:

ZENOSS_MODEL_METADATA_TAGS=source-type name simpleCustomRelationshipSourceTag simpleCustomRelationshipSinkTag

Those are the only required server configuration changes.

Update and restart the sample application

In the terminal session where the sample-app.sh script is running in the Alpine container, enter CTRL-c.

You'll update it and then restart it in a bit.
In the terminal session where the gostatsd server is running, restart the server.
1. Enter CTRL-c to stop the server.
2. Start the server again, to pick up the new configuration.
```
sudo docker run --rm --name zenoss_gostatsd --env-file ${HOME}/myvars --net mynet -ti zenoss/gostatsd:latest | grep -E 'sample-app|ERRO'
```

In the Alpine container, use a text editor to update the script.

The script needs to send the relationship sink tag with every data point. Here's the updated script:

#!/bin/sh
RANDOM=$$
SOURCE_TYPE="source-type:com.atlassian.gostatsd.tutorial.sample-app"
SOURCE="source:statsd.tutorial.sample-app"
MY_LOCAL_PART="<MyLocalPart>"
NAME="name:<MyLocalPart>.sample-app"
SINK="simpleCustomRelationshipSinkTag:statsd.tutorial.<MyLocalPart>"
while :
do
  echo "random.count:${RANDOM}|c|#${SOURCE_TYPE},${SOURCE},${MY_LOCAL_PART},${NAME},${SINK}" | nc -u -w1 zenoss_gostatsd 8125
  echo "random.number:${RANDOM}|g|#${SOURCE_TYPE},${SOURCE},${MY_LOCAL_PART},${NAME},${SINK}" | nc -u -w1 zenoss_gostatsd 8125
  echo -n "."
  sleep 3
done

Remember to replace <MyLocalPart>.

Restart the script:
```
/tmp/sample-app.sh
```

Test the result

In Zenoss Cloud, open the dashboard where the tutorial metrics are displayed.

After a few minutes, click the link named <MyLocalPart>.sample-app in the MultiMetric tile.

In Smart View, you see the sample application entity on the left side, and the server entity on the right. If you do not, wait a little while longer for the relationship to propagate.

--

Thank you for completing the tutorial!

To learn more about using the StatsD agent, see the Zenoss application examples repository.