How PagerDuty and Chronosphere work together to help you get alerted faster

on November 18th 2021

One of the most important capabilities of an observability platform is alerting. How quickly can you know when something is wrong, so you can rapidly triage and remediate that problem? Chronosphere recently released a new approach to defining alerts called “Monitors,” which gives users more flexibility with alerts and makes them easier to create and manage.

An alert is only useful if it’s seen quickly and by the right on-call team, and that’s where PagerDuty comes in. Many of our customers use Chronosphere and PagerDuty together to:

  • Know sooner: Rapidly notify the on-call engineers with alerts from Chronosphere. Chronosphere generates alerts on what you define, meaning notifications about important events are in your hands.
  • Cut through the noise: Tailor the severity, team, and notification channel for alerts generated by Chronosphere Monitors based on a wide variety of applications and metadata.
  • Triage incidents faster: When you use Chronosphere and PagerDuty together, on-call engineers get the context they need in a seamless fashion, allowing them to quickly jump into remediation mode.

How the integration works

Chronosphere uses the concept of “Monitors” to watch time series data, and generate an alert when that time series violates a specified condition. When an alert triggers it sends a notification to endpoints you specify (called a “notifier”), including PagerDuty. The notification contains the data from the triggering time series and any meta data added to the monitor.

You can trigger notifications to different PagerDuty services, or group alerts into exciting incidents.

Once the time series data returns to a value within the specified condition you can optionally send a “resolved” notification to PagerDuty.

An image showing the Chronosphere Monitors feature and a zoomed in view of the notifications section.

Requirements

  • PagerDuty integrations require an Admin base role for account authorization. If you do not have this role, please reach out to an Admin or Account Owner within your organization to configure the integration.
  • You need to use Chronosphere (UI or CLI) with a full “Service account” and not a “Restricted Service Account”.

Support

If you need help with this integration, please contact customer-support@chronosphere.io

Integration walkthrough

In PagerDuty

There are two ways to integrate with PagerDuty: via global event routing or directly through an integration on a PagerDuty service. Integrate with global event routing if you want to build different routing rules based on the events coming from Chronosphere. Integrate with a PagerDuty service if you don’t need to route alerts from the Chronosphere to different responders based on the event payload. 

An image showing the Default Global Request ruleset in PagerDuty.

Integrating with Global Event Routing

  1. From the Automation menu, select Event Rules and click your Default Global Ruleset.
  2. On the Event Rules screen, copy your Integration Key.
  3. Continue to the “In Chronosphere” section below.

Integrating with a PagerDuty service

  1. From the Services menu, select Service Directory.
  2. If you are adding the integration to an existing service, click the name of the service you want to add the integration to. Then select the Integrations tab and click Add a new integration.
    1. If you are creating a new service for the integration, read the Configuring Services and Integrations documentation, and follow the steps outlined in the “Create a New Service” section.
  3. Select “Prometheus”from the Integrations list and add it to the service, this redirects you to the Integrations tab.
An image showing selecting the Prometheus integration.
  1. Copy the Integration Key of the service
An image showing where to find the Integration Key for a service.

In Chronosphere

Chronosphere calls the endpoints you can send notifications to a “notifier”. To setup a PagerDuty notifier, open the Services menu in PagerDuty, and click the + New Service button.

You can create the notifier with the Chronosphere UI from the Settings > Notifiers menu. Click the + Create Notifier button, and select the PagerDuty option. Use the value of the Integration Key for the Service Key field and  “https://events.pagerduty.com/v2/enqueue” for the URL field.

An image showing how to create a PagerDuty Notifier with Chronosphere.

You can define the notifier(s) in a YAML file and use Chronosphere’s CLI tool to create them. Use the value of the Integration Key for the service_key field and  “https://events.pagerduty.com/v2/enqueue” for the url field. If you want to resolve the alert in PagerDuty when it resolves in Chronosphere, set send_resolved to “true” in the base_config.

api_version: v2
kind: Notifier
spec:
  notifier:
    name: team-pagerduty
    slug: team-pagerduty
    pagerduty:
    - base_config:
        # Whether or not to notify about resolved alerts.
        send_resolved: true
      service_key: <service_key>
      url: https://events.pagerduty.com/v2/enqueue
      severity: critical

Create the notifier using the CLI tool with the chronoctl apply -f file.yaml command.

You can define the notifier(s) in a .tf file and use Chronosphere Terraform provider to create them. Use the value of the Integration Key for the service_key field and  “https://events.pagerduty.com/v2/enqueue” for the url field. If you want to resolve the alert in PagerDuty when it resolves in Chronosphere, set send_resolved to true.

resource "chronosphere_pagerduty_alert_notifier" "default" {
  name = "Pagerduty Notifier"

  # Notifier-specific required configuration
  # Detailed definitions can be found at: https://developer.pagerduty.com/docs/events-api-v2/trigger-events/
  severity      = "info"
  url           = "https://events.pagerduty.com/v2/enqueue"

  send_resolved = true
}

Create the notifier using the provider with the terraform apply command.

When you create a notifier with the Chronosphere CLI tool or Terraform, you can send annotations to PagerDuty containing extra information, such as a direct link to the originating source.

For the CLI tool YAML file add something like the following:

api_version: v2
kind: Receiver
spec:
 receiver:
   email: null
   name: team-pagerduty
   opsGenie: null
   pagerduty:
     - base_config:
         send_resolved: true
       images:
         - src: "https://img.stackshare.io/service/12253/default_c4c3e2b994306c3a6028218ea49291a775fc3275.png"
           alt: "Chronosphere"
           href: "https://chronosphere.io"
       links:
         - href: "https://chronosphere.io"
           text: "Chronosphere"
         - href: "https://chronosphere.io"
           text: "Chronosphere"
       service_key: R02EU5OAWHP67AENFIGGBOO580OO2CXS
       severity: critical
       url: https://events.pagerduty.com/v2/enqueue
   slack: null
   webhook: null

And for the Terraform provider:

 image {
    # src is required for every image
    src  = "https://img.stackshare.io/service/12253/default_c4c3e2b994306c3a6028218ea49291a775fc3275.png"
    alt  = "Chronosphere"
    href = ""
  }
  link {
    # href is required for every link
    href = "https://chronosphere.io/"
    text = "some text"
  }

Chronosphere uses “Buckets” as a container for storing monitors, dashboards, and metrics. Every monitor, dashboard, and metric must belong to exactly one bucket. When you create (or edit) a bucket, you define the notifier policies, and there you can select the PageDuty notifier(s) you created.

An image showing setting notification policies.

How to uninstall

To remove the Chronosphere notifier, you first need to change the notification policy of any Buckets that use it. 

To delete with the UI, then open the Settings > Notifiers menu, click the notifier, and the delete icon.

An image showing the delete button for a notifier.

To delete with the terraform provider, remove the notifier definition from the TF file and use the terraform apply command.

To delete with the CLI tools, use the chronoctl delete notifier -n notifier-name command.

Other resources you may be interested in

Interested in what we are building?