Skip to content

About streaming data

Zenoss Cloud provides several methods for gathering model, metric, and event data without a Collection Zone. Each method uses the data receiver resources of the Zenoss API to update Zenoss Cloud. Updates are either a single transaction or a regular flow of data, or streaming data.

Data collection methods include:

  • ZenPacks. Learn more.

  • Connectors. A connector gathers data from public cloud applications with just a little configuration.

  • Agents. An independent agent sends data either directly to Zenoss Cloud or indirectly, through a Zenoss Data Monitor proxy. Agents are distributed as Docker images or platform-native packages.

  • Integrations. You can create customized applications with data receiver resources or with existing Zenoss libraries.

All your streaming data flows can be customized with the policy service, which allows you to define, manage, and enforce policies that control access to your resources. For more information, see Policy service.

Connectors

Zenoss Cloud provides ready-made connectors for public cloud applications. To enable monitoring, you need only configure a few fields—no agent installation is required. Supported connectors:

  • Microsoft 365 supports Microsoft Teams, Microsoft OneDrive, Microsoft Exchange, and Microsoft SharePoint.
  • Site Availability Monitor supports monitoring the uptime of your cloud-based web applications.
  • Zoom supports the Zoom platform.
  • Kubernetes supports clusters on AWS, Azure, and Google Cloud, as well as self-hosted clusters.

Before setting up your connectors

Depending on your setup, you might need to know the source IP address from which to expect connections. For example, if your connector is monitoring services behind a firewall, then you might want to configure those services to allow access from the source IP address. These source IP addresses are critical for your Kubernetes and Site Availability Monitoring connectors, but can help when troubleshooting Microsoft and Zoom connection issues as well.

Therefore, you might want to add the following IP addresses to your IP allowlist. If needed, check with your Zenoss support associate to identify your Zenoss region.

  • US Central: 34.69.219.244
  • US West: 34.125.204.146
  • Europe: 35.246.162.90
  • Australia: 34.87.243.148

Agents

Zenoss provides the following purpose-built clients, which stream data to Zenoss Cloud through a standard HTTPS connection:

Agents are distributed either as Docker images or as platform-native packages.

Zenoss Data Monitor agent

The Zenoss Data Monitor agent is a modular monitoring program that runs as a background process. It collects data from Linux or Windows hosts and streams it directly to Zenoss Cloud. For more information, see About Zenoss Data Monitor.

Kubernetes agent

The Zenoss agent for Kubernetes is a monitoring program that runs as a background process. It collects key cluster metrics from kube-apiserver and then streams them directly to Zenoss Cloud. The agent sends both metric and model data for pods, containers, nodes, namespaces, and the cluster itself. The metric data can be viewed in dashboards or in Smart View; a Kubernetes dashboard template is available. The model data includes dependency relationships, which enable Smart View analyses of related cluster entities.

The Zenoss agent for Kubernetes supports Kubernetes 1.10 through 1.16 and requires metrics-server. You include the agent in a cluster with a deployment and Kubernetes schedules the agent in its own pod, on one node. The size of a cluster determines how much RAM the agent consumes; in small clusters, it consumes approximately 15MB. The agent uses incremental change notifications to collect data, rather than a polling interval, and so receives updates any time a monitored property changes. The agent sends a batch of metric data to Zenoss Cloud every 60 seconds and sends a batch of model data when the model data changes. The source code of the agent is public and an image containing its binary is available on Docker Hub.

Integrations

You can create customized applications with data receiver resources or with existing Zenoss libraries. Use the Zenoss API to configure the resources of the data receiver service to send entities, metric data points, and events directly to Zenoss Cloud.

Policy service

The policy service provides centralized data and operations management through declarations rather than code.

  • Transform incoming data with ingest policies.

    Separate metadata from data in incoming metric data streams to create distinct entities and metrics, without writing customized ETL scripts. Or add metadata to event data streams, to categorize events or facilitate action service processing.

  • Identify entity types with entity policies.

    Provide a tag for instances of an entity type, or standardize metadata from different sources so that, for example, all Kubernetes clusters have the same tags and can be easily found.

  • Manage anomaly detection with anomaly policies.

    Add or subtract metrics from the list of metrics for which the anomaly detection service trains models.

A datasource associates an incoming stream of data with one or more policies. Policies are not used unless they are included in a datasource.

When to customize

You do not need to customize any of the default policies that Zenoss provides to monitor your environment. The defaults have been tested extensively in production and work as designed.

  • You may wish to customize an anomaly policy to add a metric to the anomaly detection service. You are welcome to do so at any time.

  • If you wish to integrate an application or add a new, custom datasource, please contact Zenoss Support to arrange assistance creating a custom ingest or entity policy.