Prometheus alertmanager

Alertmanager is the official service prometheus communicates with to issue alerts.

Documentation

repo	https://github.com/prometheus/alertmanager
official docs	https://prometheus.io/docs/alerting/latest/configuration/
sample config	https://github.com/prometheus/alertmanager#example
template docs	https://prometheus.io/docs/prometheus/latest/configuration/template_examples/

Locations

9093	default port that receives alerts
${PREFIX}/etc/alertmanager/alertmanager.yml	config

Notes

alertmanager install

alertmanager configuration

alertmanager configurations

alertmanager usage

Configuration

AlertManager

AlertManager can issue notifications using various methods.
See docs for all options (ex. email, http, slack, wechat, ...)

TODO:

to reduce nesting, document "General" only, and create a section for all-encompassing examples (ex: SMTP w/ postfix)

General

Sections

global:          # general settings
templates:       # configure template locations (templates for alert messages)
route:           # root-route, where alerts enter
inhibit_rules:   # rules to mute alerts, when other alerts are already firing
receivers:       # alerts are issued to receivers

Routes

route:
  receiver: team-X-mails              # default receiver for all routes
  group_by: ['cluster', 'alertname']  # alerts batched by labels. one alert fired per-batch at a time.
  repeat_interval: 3h                 # re-issue alert after this time-interval if not resolved

  # optionally, you can match on alert-labels
  # and alter the alert/receiver
  # (can be nested for gradually more specific rules)
  routes:
    - matchers:
      - service=~"ha|nginx|wsgi"      # if alert's label matches regex, issue to this receiver
      receiver: team-X-mails
      routes:                         # (opt) child-routes w/ extra label-matches
        - matchers:                   #       (ex: higher severity alert)
          - severity="critical"
          receiver: team-X-pager
    - matchers:
      # ...

SMTP

First, setup a send-only postfix install on the localhost.

# /usr/local/etc/alertmanager/alertmanager.yml

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

route:
  receiver: smtp-local

receivers:
  - name: 'smtp-local'
    email_configs:
    - to: 'to@domain.com'
      from: 'foo@domain.com'
      require_tls: false
      smarthost: localhost:25
      send_resolved: true

HTTP

Prometheus

# /usr/local/etc/prometheus.yml

alerting:
  alertmanagers:
    - static_configs:
      - targets: ['localhost:9093']

Usage

amtool check-config alertmanager.yml  # validate an alertmanager config