Prometheus alertmanager

Alertmanager is the official service prometheus communicates with to issue alerts.

Documentation

repo	https://github.com/prometheus/alertmanager
official docs	https://prometheus.io/docs/alerting/latest/configuration/
sample config	https://github.com/prometheus/alertmanager#example
template docs	https://prometheus.io/docs/prometheus/latest/configuration/template_examples/

Locations

9093	default port that receives alerts
${PREFIX}/etc/alertmanager/alertmanager.yml	config

Install

pkg install alertmanager  # FreeBSD
pacman -S alertmanager    # Archlinux

You'll also need to enable/start the service.

Configuration

AlertManager

AlertManager can issue notifications using various methods.
See docs for all options

General

Sections

global:          # general settings
templates:       # configure template locations (templates for alert messages)
route:           # root-route, where alerts enter
inhibit_rules:   # rules to mute alerts, when other alerts are already firing
receivers:       # alerts are issued to receivers

routes

route:
  receiver: team-X-mails              # default receiver for all routes
  group_by: ['cluster', 'alertname']  # alerts batched by labels. one alert fired per-batch at a time.
  repeat_interval: 3h                 # re-issue alert after this time-interval if not resolved
  routes:
    - matchers:
      - service=~"ha|nginx|wsgi"      # if alert's label matches regex, issue to this receiver
      receiver: team-X-mails
      routes:                         # (opt) child-routes w/ extra label-matches
        - matchers:                   #       (ex: higher severity alert)
          - severity="critical"
          receiver: team-X-pager
    - matchers:
      # ...

SMTP

First, setup a send-only postfix install on the localhost.

# /usr/local/etc/alertmanager/alertmanager.yml

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'

route:
  receiver: smtp-local

receivers:
  - name: 'smtp-local'
    email_configs:
    - to: 'to@domain.com'
      from: 'foo@domain.com'
      require_tls: false
      smarthost: localhost:25
      send_resolved: true

HTTP

Prometheus

# /usr/local/etc/prometheus.yml

alerting:
  alertmanagers:
    - static_configs:
      - targets: ['localhost:9093']

Usage

amtool check-config alertmanager.yml  # validate an alertmanager config