Alertmanager configuration

Locations

${PREFIX}/etc/alertmanager/alertmanager.yml	alertmanager config
${PREFIX}/prometheus.yml	prometheus config

Prometheus

Prometheus is responsible for creating the rules that issue the alerts.

# /usr/local/etc/prometheus.yml

alerting:
  alertmanagers:
    - static_configs:
      - targets: ['localhost:9093']

AlertManager

Alertmanager is responsible for consuming fired-alerts, and deciding who/how/if to alert.

Overview

AlertManager can issue notifications using various methods.
See docs for all options (ex. email, http, slack, wechat, ...)

global:          # general settings
route:           # root-route, where alerts enter
receivers:       # alerts are issued to receivers

templates:       # configure template locations (templates for alert messages)
inhibit_rules:   # rules to mute alerts, when other alerts are already firing

Routes

Routes determine when/how notifications are sent, and who they are sent to.

route:
  receiver: team-X-mails              # default receiver for all routes
  group_by: ['cluster', 'alertname']  # alerts batched by labels. one alert fired per-batch at a time.
  repeat_interval: 3h                 # re-issue alert after this time-interval if not resolved

  # optionally, you can match on alert-labels
  # and alter the alert/receiver
  # (can be nested for gradually more specific rules)
  routes:
    - matchers:
      - service=~"ha|nginx|wsgi"      # if alert's label matches regex, issue to this receiver
      receiver: team-X-mails
      routes:                         # you can recursively nest routes, for more specific label matches
        - matchers:
          - severity="critical"
          receiver: team-X-pager
    - matchers:
      # ...

Receivers

Receivers are used in routes, and represent a communication-method for issuing an alert.
ex: email, slack, pagerduty, ...

receivers:
  # send email
  - name: 'team-X-mails'
    email_configs:
      - to: 'team-X+alerts@example.org'

  # send email AND page w/ pagerduty
  - name: 'team-X-pager'
    email_configs:
      - to: 'team-X+alerts-critical@example.org'
    pagerduty_configs:
      - service_key: 'abcdefg'