Prometheus alertmanager: Difference between revisions

From wikinotes
No edit summary
 
(23 intermediate revisions by the same user not shown)
Line 11: Line 11:
| sample config || https://github.com/prometheus/alertmanager#example
| sample config || https://github.com/prometheus/alertmanager#example
|-
|-
| template docs || https://prometheus.io/docs/prometheus/latest/configuration/template_examples/
|}
|}
</blockquote><!-- Documentation -->
</blockquote><!-- Documentation -->


= Locations =
= Tutorials =
<blockquote>
{| class="wikitable"
|-
| grafana/pagerduty tutorial || https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/
|-
|}
</blockquote><!-- Tutorials -->
 
= Routes =
<blockquote>
<blockquote>
{| class="wikitable"
{| class="wikitable"
Line 20: Line 30:
| <code>9093</code> || default port that receives alerts
| <code>9093</code> || default port that receives alerts
|-
|-
| <code>${PREFIX}/etc/alertmanager/alertmanager.yml</code> || config
| <code>http://localhost:9093</code> || alertmanager ui
|-
|-
|}
|}
</blockquote><!-- Locations -->
</blockquote><!-- Routes -->


= Install =
= Locations =
<blockquote>
<blockquote>
<syntaxhighlight lang="bash">
{| class="wikitable"
pkg install alertmanager # FreeBSD
|-
pacman -S alertmanager    # Archlinux
| <code>${PREFIX}/etc/alertmanager/alertmanager.yml</code> || alertmanager config
</syntaxhighlight>
|-
| <code>${PREFIX}/etc/prometheus.yml</code> || prometheus config
|-
| <code>/var/log/alertmanager.log</code> || default-log (unless alt syslog service - see [[alertmanager install]].)
|-
|  <code>/var/db/alertmanager</code> || data dir
|-
|}
</blockquote><!-- Locations -->


You'll also need to enable/start the service.
= Notes =
</blockquote><!-- Install -->
 
= Configuration =
<blockquote>
<blockquote>
== AlertManager ==
{| class="wikitable"
<blockquote>
|-
AlertManager can issue notifications using various methods.<br>
| [[alertmanager install]]
See [https://prometheus.io/docs/alerting/latest/configuration/#configuration-file docs] for all options
|-
| [[alertmanager configuration]]
|-
| [[alertmanager configurations]]
|-
| [[alertmanager usage]]
|-
| [[alertmanager debugging]]
|-
|}
</blockquote><!-- Notes -->


=== General ===
= Usage =
<blockquote>
<blockquote>
Sections
<syntaxhighlight lang="bash">
<syntaxhighlight lang="yaml">
amtool check-config alertmanager.yml  # validate an alertmanager config
global:          # general settings
templates:      # configure template locations (templates for alert messages)
route:          # root-route, where alerts enter
inhibit_rules:  # rules to mute alerts, when other alerts are already firing
receivers:      # alerts are issued to receivers
</syntaxhighlight>
</syntaxhighlight>


<syntaxhighlight lang="yaml">
trigger a fake alert ([https://blog.mafr.de/2020/09/13/testing-alertmanager/ source])
route:
<syntaxhighlight lang="bash">
  group_by:
#! /usr/bin/env sh
  group_interval:
  repeat_interval:
</syntaxhighlight>
</blockquote><!-- General -->
 
=== SMTP ===
<blockquote>
First, setup a send-only [[postfix]] install on the localhost.
 
<syntaxhighlight lang="yaml">
# /usr/local/etc/alertmanager/alertmanager.yml


global:
URL="http://localhost:9093/api/v1/alerts"
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@example.org'
  smtp_auth_username: 'alertmanager'
  smtp_auth_password: 'password'


route:
curl -si -X POST -H "Content-Type: application/json" "$URL" -d '
  receiver: smtp-local
[
 
  {
receivers:
     "labels": {
  - name: 'smtp-local'
       "alertname": "InstanceDown",
    email_configs:
       "instance": "localhost:8080",
     - to: 'to@domain.com'
       "job": "node",
       from: 'foo@domain.com'
       "severity": "critical"
       require_tls: false
    },
       smarthost: localhost:25
    "annotations": {
       send_resolved: true
      "summary": "Instance is down"
</syntaxhighlight>
     },
</blockquote><!-- SMTP -->
    "generatorURL": "http://localhost:9090/graph"
 
  }
=== HTTP ===
]
<blockquote>
'
 
</blockquote><!-- HTTP -->
</blockquote><!-- AlertManager -->
 
== Prometheus ==
<blockquote>
<syntaxhighlight lang="yaml">
# /usr/local/etc/prometheus.yml
 
alerting:
  alertmanagers:
     - static_configs:
      - targets: ['localhost:9093']
</syntaxhighlight>
</blockquote><!-- Prometheus -->
</blockquote><!-- Configuration -->
 
= Usage =
<blockquote>
<syntaxhighlight lang="bash">
amtool check-config alertmanager.yml  # validate an alertmanager config
</syntaxhighlight>
</syntaxhighlight>
</blockquote><!-- Usage -->
</blockquote><!-- Usage -->

Latest revision as of 21:09, 20 February 2022

Alertmanager is the official service prometheus communicates with to issue alerts.

Documentation

repo https://github.com/prometheus/alertmanager
official docs https://prometheus.io/docs/alerting/latest/configuration/
sample config https://github.com/prometheus/alertmanager#example
template docs https://prometheus.io/docs/prometheus/latest/configuration/template_examples/

Tutorials

grafana/pagerduty tutorial https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/

Routes

9093 default port that receives alerts
http://localhost:9093 alertmanager ui

Locations

${PREFIX}/etc/alertmanager/alertmanager.yml alertmanager config
${PREFIX}/etc/prometheus.yml prometheus config
/var/log/alertmanager.log default-log (unless alt syslog service - see alertmanager install.)
/var/db/alertmanager data dir

Notes

alertmanager install
alertmanager configuration
alertmanager configurations
alertmanager usage
alertmanager debugging

Usage

amtool check-config alertmanager.yml  # validate an alertmanager config

trigger a fake alert (source) 

#! /usr/bin/env sh

URL="http://localhost:9093/api/v1/alerts"

curl -si -X POST -H "Content-Type: application/json" "$URL" -d '
[
  {
    "labels": {
      "alertname": "InstanceDown",
      "instance": "localhost:8080",
      "job": "node",
      "severity": "critical"
    },
    "annotations": {
      "summary": "Instance is down"
    },
    "generatorURL": "http://localhost:9090/graph"
  }
]
'
Retrieved from "http://willpittman.net:8080/index.php?title=Prometheus_alertmanager&oldid=29071"