Prometheus alertmanager: Difference between revisions

From wikinotes
No edit summary
 
(14 intermediate revisions by the same user not shown)
Line 15: Line 15:
</blockquote><!-- Documentation -->
</blockquote><!-- Documentation -->


= Locations =
= Tutorials =
<blockquote>
{| class="wikitable"
|-
| grafana/pagerduty tutorial || https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/
|-
|}
</blockquote><!-- Tutorials -->
 
= Routes =
<blockquote>
<blockquote>
{| class="wikitable"
{| class="wikitable"
Line 21: Line 30:
| <code>9093</code> || default port that receives alerts
| <code>9093</code> || default port that receives alerts
|-
|-
| <code>${PREFIX}/etc/alertmanager/alertmanager.yml</code> || config
| <code>http://localhost:9093</code> || alertmanager ui
|-
|-
|}
|}
</blockquote><!-- Locations -->
</blockquote><!-- Routes -->


= Install =
= Locations =
<blockquote>
<blockquote>
<syntaxhighlight lang="bash">
{| class="wikitable"
pkg install alertmanager # FreeBSD
|-
pacman -S alertmanager    # Archlinux
| <code>${PREFIX}/etc/alertmanager/alertmanager.yml</code> || alertmanager config
</syntaxhighlight>
|-
| <code>${PREFIX}/etc/prometheus.yml</code> || prometheus config
|-
| <code>/var/log/alertmanager.log</code> || default-log (unless alt syslog service - see [[alertmanager install]].)
|-
|  <code>/var/db/alertmanager</code> || data dir
|-
|}
</blockquote><!-- Locations -->


You'll also need to enable/start the service.
= Notes =
</blockquote><!-- Install -->
 
= Configuration =
<blockquote>
<blockquote>
== AlertManager ==
{| class="wikitable"
<blockquote>
|-
AlertManager can issue notifications using various methods.<br>
| [[alertmanager install]]
See [https://prometheus.io/docs/alerting/latest/configuration/#configuration-file docs] for all options (ex. email, http, slack, wechat, ...)
|-
| [[alertmanager configuration]]
|-
| [[alertmanager configurations]]
|-
| [[alertmanager usage]]
|-
| [[alertmanager debugging]]
|-
|}
</blockquote><!-- Notes -->


=== General ===
= Usage =
<blockquote>
<blockquote>
 
<syntaxhighlight lang="bash">
==== Sections ====
amtool check-config alertmanager.yml  # validate an alertmanager config
<syntaxhighlight lang="yaml">
global:          # general settings
templates:      # configure template locations (templates for alert messages)
route:          # root-route, where alerts enter
inhibit_rules:  # rules to mute alerts, when other alerts are already firing
receivers:      # alerts are issued to receivers
</syntaxhighlight>
</syntaxhighlight>


==== Routes ====
trigger a fake alert ([https://blog.mafr.de/2020/09/13/testing-alertmanager/ source])
<syntaxhighlight lang="yaml">
<syntaxhighlight lang="bash">
route:
#! /usr/bin/env sh
  receiver: team-X-mails              # default receiver for all routes
  group_by: ['cluster', 'alertname']  # alerts batched by labels. one alert fired per-batch at a time.
  repeat_interval: 3h                # re-issue alert after this time-interval if not resolved
 
  # optionally, you can match on alert-labels
  # and alter the alert/receiver
  # (can be nested for gradually more specific rules)
  routes:
    - matchers:
      - service=~"ha|nginx|wsgi"      # if alert's label matches regex, issue to this receiver
      receiver: team-X-mails
      routes:                        # (opt) child-routes w/ extra label-matches
        - matchers:                  #      (ex: higher severity alert)
          - severity="critical"
          receiver: team-X-pager
    - matchers:
      # ...
</syntaxhighlight>
</blockquote><!-- General -->
 
=== SMTP ===
<blockquote>
First, setup a send-only [[postfix]] install on the localhost.
 
<syntaxhighlight lang="yaml">
# /usr/local/etc/alertmanager/alertmanager.yml
 
global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'
 
route:
  receiver: smtp-local
 
receivers:
  - name: 'smtp-local'
    email_configs:
    - to: 'to@domain.com'
      from: 'foo@domain.com'
      require_tls: false
      smarthost: localhost:25
      send_resolved: true
</syntaxhighlight>
</blockquote><!-- SMTP -->


=== HTTP ===
URL="http://localhost:9093/api/v1/alerts"
<blockquote>


</blockquote><!-- HTTP -->
curl -si -X POST -H "Content-Type: application/json" "$URL" -d '
</blockquote><!-- AlertManager -->
[
 
  {
== Prometheus ==
    "labels": {
<blockquote>
      "alertname": "InstanceDown",
<syntaxhighlight lang="yaml">
      "instance": "localhost:8080",
# /usr/local/etc/prometheus.yml
      "job": "node",
 
      "severity": "critical"
alerting:
    },
  alertmanagers:
     "annotations": {
     - static_configs:
       "summary": "Instance is down"
       - targets: ['localhost:9093']
    },
</syntaxhighlight>
    "generatorURL": "http://localhost:9090/graph"
</blockquote><!-- Prometheus -->
  }
</blockquote><!-- Configuration -->
]
 
'
= Usage =
<blockquote>
<syntaxhighlight lang="bash">
amtool check-config alertmanager.yml  # validate an alertmanager config
</syntaxhighlight>
</syntaxhighlight>
</blockquote><!-- Usage -->
</blockquote><!-- Usage -->

Latest revision as of 21:09, 20 February 2022

Alertmanager is the official service prometheus communicates with to issue alerts.

Documentation

repo https://github.com/prometheus/alertmanager
official docs https://prometheus.io/docs/alerting/latest/configuration/
sample config https://github.com/prometheus/alertmanager#example
template docs https://prometheus.io/docs/prometheus/latest/configuration/template_examples/

Tutorials

grafana/pagerduty tutorial https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/

Routes

9093 default port that receives alerts
http://localhost:9093 alertmanager ui

Locations

${PREFIX}/etc/alertmanager/alertmanager.yml alertmanager config
${PREFIX}/etc/prometheus.yml prometheus config
/var/log/alertmanager.log default-log (unless alt syslog service - see alertmanager install.)
/var/db/alertmanager data dir

Notes

alertmanager install
alertmanager configuration
alertmanager configurations
alertmanager usage
alertmanager debugging

Usage

amtool check-config alertmanager.yml  # validate an alertmanager config

trigger a fake alert (source) 

#! /usr/bin/env sh

URL="http://localhost:9093/api/v1/alerts"

curl -si -X POST -H "Content-Type: application/json" "$URL" -d '
[
  {
    "labels": {
      "alertname": "InstanceDown",
      "instance": "localhost:8080",
      "job": "node",
      "severity": "critical"
    },
    "annotations": {
      "summary": "Instance is down"
    },
    "generatorURL": "http://localhost:9090/graph"
  }
]
'
Retrieved from "http://willpittman.net:8080/index.php?title=Prometheus_alertmanager&oldid=29071"