Prometheus alertmanager

Alertmanager is the official service prometheus communicates with to issue alerts.

Documentation

repo	https://github.com/prometheus/alertmanager
official docs	https://prometheus.io/docs/alerting/latest/configuration/
sample config	https://github.com/prometheus/alertmanager#example
template docs	https://prometheus.io/docs/prometheus/latest/configuration/template_examples/

Tutorials

grafana/pagerduty tutorial

https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/

Routes

9093	default port that receives alerts
http://localhost:9093	alertmanager ui

Locations

${PREFIX}/etc/alertmanager/alertmanager.yml	alertmanager config
${PREFIX}/etc/prometheus.yml	prometheus config
/var/log/alertmanager.log	default-log (unless alt syslog service - see alertmanager install.)
/var/db/alertmanager	data dir

Notes

alertmanager install

alertmanager configuration

alertmanager configurations

alertmanager usage

alertmanager debugging

Usage

amtool check-config alertmanager.yml  # validate an alertmanager config

trigger a fake alert (source)

#! /usr/bin/env sh

URL="http://localhost:9093/api/v1/alerts"

curl -si -X POST -H "Content-Type: application/json" "$URL" -d '
[
  {
    "labels": {
      "alertname": "InstanceDown",
      "instance": "localhost:8080",
      "job": "node",
      "severity": "critical"
    },
    "annotations": {
      "summary": "Instance is down"
    },
    "generatorURL": "http://localhost:9090/graph"
  }
]
'