Alertmanager configurations: pagerduty

Push alerts from prometheus alertmanager to pagerduty.

Documentation

prometheus docs	https://prometheus.io/docs/alerting/latest/configuration/#pagerduty_config
pagerduty docs	https://www.pagerduty.com/docs/guides/prometheus-integration-guide/

Overview

There are a couple of choices for integrations

requirements	AlertManager Key Name	PagerDuty Integration Type	PagerDuty API version
alertmanager-0.11+	routing_key	Events API v2	Events API v2
alertmanager (any)	service_key	Prometheus	Events API v1

Prometheus/Alertmanager

WARNING:

Despite the warning about the use of routing_key for the Events API v2, only service_key has worked for me

# /usr/local/etc/alertmanager/alertmanager.yml

global:
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'  # Events API v2

route:
  receiver: smtp              # use smtp if no other rule matches
  routes:
    - receiver: 'pagerduty'   # use pagerduty if the label 'severity' matches 'page'
      matchers:
        - 'severity = page'

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: ${INTEGRATION_KEY}  # Events API v2

  - name: 'smtp'
    # ...

PagerDuty

For some reason, Events API V2 did not show up when trying to add integrations to an existing service.
It did for a new service though.

- Create a service in pagerduty
  - Tab: Services
    - New Service
      - Name: # whatever
      - Assign: # whatever
      - Reduce Noise: # whatever
      - Integrations:
        - [x] Events API V2

# Copy the integration key from this integration