OpenTelemetry

OpenTelemetry, also known as OTel, is an open source, observability framework that helps IT teams collect, process, and export telemetry data, including metrics, so that they can better understand and troubleshoot their IT applications and infrastructure. Because OpenTelemetry is both vendor-neutral and tool-neutral, you can use it with various observability backends like Zenoss Cloud. Zenoss Cloud can help ease the pain of monitoring different infrastructure pieces using various technologies, like OTeL, SNMP, WBEM, SMI-S, and other vendor specific technologies.

OpenTelemetry enables observability through collecting, processing and exporting signals, like metrics, that provide a look at the underlying activity of the operating system and applications running on a platform. Because metrics can capture a measurement, like memory usage, of a component or service at a point in time, you can analyze not just the measurement itself, but also the time at which it was captured and the associated metric metadata known as attributes in OpenTelemetry.

In Zenoss, you can integrate with OpenTelemetry to collect sum, gauge, and histogram metrics to gain insights of the behavior of your applications. When OpenTelemetry is integrated with Zenoss, developers can leverage existing metric analysis tools in Zenoss while benefiting from enhanced observability capabilities.

Zenoss leverages the following key OpenTelemetry components:

OTLP specification for defining the encoding of telemetry data and the protocol used to exchange data between the client and the server.
Collectors for receiving, processing and exporting telemetry data collected from instrumented applications. They collect data from multiple sources, process it, then export the processed data to back-end storage or analysis systems.

Getting Started

The OpenTelemetry sample tutorial guides you through setting up the OpenTelemetry Collector, a proxy that receives, processes, and exports telemetry data for you. Exporting OpenTelemetry metrics into Zenoss Cloud uses the standard OpenTelemetry Protocol (OTLP). Most systems that support exporting OpenTelemetry data also support the OTLP protocol.

To get started, check that you meet the following prerequisites and then configure your application for exporting metrics using the OpenTelemetry Protocol (OTLP).

Prerequisites

Before getting started with OpenTelemetry, verify that you have the the proper permissions and a Zenoss API key:

Make sure your account is assigned the Key Administrator or Manager role.
Locate or generate your Zenoss API key. Ask your Zenoss administrator for your Zenoss API key or generate a key.

Configure your application for exporting metrics

Configure your application to use the OpenTelemetry Protocol (OTLP) for exporting telemetry data.

Configure your system to export your telemetry data using the following settings:

Exporter type: OTLP
Endpoint: The sample tutorial in this guide uses https://api.zenoss.io:443, but your endpoint might differ. For more information to help you determine the correct endpoint to use, see Zenoss API endpoints.
Header: zenoss-api-key=YOUR-ZENOSS-API-KEY

OpenTelemetry sample tutorial

This sample tutorial walks you through configuring an OpenTelemetry Collector using the OpenTelemetry Collector Contrib distribution of the collector on a Linux system. Then, you will add three metrics to a dashboard in Zenoss. Finally, you will edit the configuration file to augment the data.

Follow this sample tutorial to complete the following tasks:

Set up an OpenTelemetry collector on a Linux system to export sample system.cpu.load_average metrics to Zenoss.
Create a dashboard to see how you can visualize the metrics in Zenoss.
Augment the data to separate the data into different sources so you can filter the data in your dashboards and other features.

Setting up the OpenTelemetry Collector

The OpenTelemetry Collector (otelcol) is a vendor-neutral way to receive, process, and export telemetry data. It supports receiving telemetry data in multiple formats, processing it, and then sending it to one or more backends.

The following configuration instructions assume that you have installed the OpenTelemetry collector. If you haven’t yet installed the collector, see the installation documentation.

Additionally, these instructions assume that you have the OpenTelemetry Collector Contrib distribution of the collector installed on a Linux system. If not, first access the otelcol-contrib distribution and install it on a Linux system using the installation documentation.

Configure the OpenTelemetry Collector

This example configuration is one possible simple configuration for getting metric data into Zenoss. For these instructions, you need the OpenTelemetry Collector Contrib distribution (otelcol-contrib) of the collector installed on a Linux system. The OpenTelemetry Collector Contrib is a good distribution for getting started because it contains many built-in receivers, processors, and exporters.

By default, the otelcol-contrib service looks for its configuration in the following file:

/etc/otelcol-contrib/config.yaml

Perform the following steps:

Go to the OpenTelemetry Collector configuration file located in the /etc/otelcol-contrib/ directory.
In the config.yaml file, do the following:
- Set the endpoint URL to the appropriate value.
- Replace YOUR-ZENOSS-API-KEY with a valid API key.

This file accomplishes the following steps:

Configures the receiver to collect metric data.
Sets up the processor that adds attributes (metadata) to the data
Sets up an exporter that sends the data to Zenoss Cloud
Sets up a metrics pipeline to use the receiver, processor, and exporter together.

receivers:
  # Gather system load metrics from the host.
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/README.md
  hostmetrics:
    scrapers:
      load:

processors:
  # Automatically add attributes such as host.name and os.type.
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md
  resourcedetection:
    detectors: [system]

exporters:
  otlp/zenoss:
    endpoint: "http://api.zenoss.io:443"
    headers:
      zenoss-api-key: "YOUR-ZENOSS-API-KEY"

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection]
      exporters: [otlp/zenoss]

Note

The OpenTelemetry Collector Contrib distribution has many other receivers, processors, and exporters available.

Restart the otelcol-contrib service after configuring it by running the following command:
```
sudo systemctl restart otelcol-contrib
```

Adding metrics to a Zenoss dashboard

Continuing with the sample use case, after configuring the OpenTelemetry Collector to send data from the "load" scraper of the "hostmetrics" receiver, you now have three new metrics available in Zenoss:

system.cpu.load_average.1m
system.cpu.load_average.5m
system.cpu.load_average.15m

Create a dashboard

To view these new metrics, create a new dashboard using the following steps.

Log in to Zenoss Cloud as a user with the Manager role, and then open DASHBOARDS.
Open the Dashboards panel on the left and click NEW DASHBOARD.
(Optional) In the name field, replace My Dashboard with a more descriptive name for the dashboard, such as OTel Default.
Specify the scope to include the source that you previously added. Click ADD ENTITIES, then enter the name of the source, otel-default, in the Other sources field. Click the matching otel-default from the list of other sources.
Click the checkmark to confirm, then click SAVE.

Configure a Graph tile

Add a graph tile to the dashboard and configure it to show the new metrics.

Click ADD A TILE. In the AVAILABLE TILES palette, click Graph to add it to the dashboard. The GRAPH panel opens on the left side. By default, the TILE CONFIGURATION tab is selected.
In the Entity and metric field, enter system then choose system.cpu.load_average.1m in the list that appears.
In the Aggregator dropdown list, select none. Click SAVE.

As a best practice, if you just set up the collector, you might want to focus on a very recent timeframe, such as the last hour. For more information about viewing metrics in the Graph tile for OpenTelemetry, see Graph tile configuration.

Note

If you click on a line in the graph tile, Zenoss directs you to Smart View for the metric's entity where you can see other metrics collected for the entity.

Augmenting the data

In the previous section, you added the otel-default source to your dashboard scope to show metrics from the collector. When you send more OpenTelemetry data into Zenoss, you may want to separate the data into different sources so you can easily filter the data in your dashboards and other features.

To accomplish this task, use the collector's resourceprocessor to set the resource's source attribute to any value you want.

The configuration file in the task shows the previous configuration, but this file uses resourceprocessor to add a "source=mysource" attribute to all of the data:

  # Add a "source=mysource" attribute to all resources.
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourceprocessor/README.md
  resource/source:
    attributes:
      - action: insert
        key: source
        value: "mysource"

In the following task, you will also update service.pipeline.metrics.processors to add the new processor to your pipeline:

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, resource/source]
      exporters: [otlp/zenoss]

Augment the data

Perform the following steps to augment the data:

Make the following changes in the configuration file:

receivers:
  # Gather system load metrics from the host.
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/README.md
  hostmetrics:
    scrapers:
      load:

processors:
  # Automatically add attributes such as host.name and os.type.
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md
  resourcedetection:
    detectors: [system]

  # Add a "source=mysource" attribute to all resources.
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourceprocessor/README.md
  resource/source:
    attributes:
      - action: insert
        key: source
        value: "mysource"

exporters:
  otlp/zenoss:
    endpoint: "http://api.zenoss.io:443"
    headers:
      zenoss-api-key: "YOUR-ZENOSS-API-KEY"

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resourcedetection, resource/source]
      exporters: [otlp/zenoss]

Restart the otelcol-contrib service after configuring it by running the following command:

sudo systemctl restart otelcol-contrib

You can now add mysource to any dashboard scope where you want to see this data without sending in other data received from the otel-default source.

OpenTelemetry terms in Zenoss

In some cases, Zenoss and OpenTelemetry share similar concepts, but use different terms for them. Use the following mappings to understand how OpenTelemetry data is represented in Zenoss.

resource/entity

In OpenTelemetry, resource is an entity in Zenoss. This is a one-to-one mapping with no caveats.

attributes/dimension attributes

In OpenTelemetry, data is described by attributes. Each unique combination of attributes (name-value pair) represents a unique item. For example, two resources might be alike in all ways except that they have different values for the host attribute. Therefore, these are two distinct resources.

In Zenoss, the equivalent of attributes is dimension attributes. This is different from metadata attributes that are extra attributes that do not make the item unique. Therefore, when Zenoss receives OpenTelemetry data, it treats all attributes as dimension attributes.

Additional resources

See the following resources to learn more about the key OpenTelemetry components used in this sample tutorial.