Install Alert Manager for Alert and send to Slack Channel
##1. Download Packet AlertManager
cd /opt
wget https://github.com/prometheus/alertmanager/releases/download/v0.21.0/alertmanager-0.21.0.linux-amd64.tar.gz
tar xvfz alertmanager-0.21.0.linux-amd64.tar.gz
##2. Create Slack Channel
First, we have to create slack channel to send alert to slack,
Klik + and create channel, refer to picture below
Go to Details channel and click add apps
Go to searching section and find and click Incoming Webhooks,
Click Configuration
Click add to slack,
Choose channel that we create earlier and click Add incoming,
Copy WebHooks URL, scroll down and save settings,
In the slack channel there is a notice like this,
##3. Config Alert Manager
Go to directory of Alert Manager that we extract earlier, cd alertmanager-0.21.0.linux-amd64
Create config and save, sudo vi config.yml
route:
# When a new group of alerts is created by an incoming alert, wait at
# least ‘group_wait’ to send the initial notification.
# This way ensures that you get multiple alerts for the same group that start
# firing shortly after another are batched together on the first
# notification.
group_wait: 10s
# When the first notification was sent, wait ‘group_interval’ to send a batch
# of new alerts that started firing for that group.
group_interval: 5m
# If an alert has successfully been sent, wait ‘repeat_interval’ to
# resend them.
repeat_interval: 30m
# A default receiver
receiver: “slack”
# All the above attributes are inherited by all child routes and can
# overwritten on each.
routes:
- receiver: “slack”
group_wait: 10s
match_re:
severity: error|warning
continue: true
receivers:
- name: “slack”
slack_configs:
- api_url: ‘https://your hooks slack'
send_resolved: true
channel: ‘#ops4_alert’
text: “{{ range .Alerts }}<!channel> {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}”
At the api_url red color section, paste WebHooks that we copy earlier from WebHooks settings,
Create service to run alert manager and save, sudo vi /etc/systemd/system/alert_manager.service
[Unit]
Description=Alert Manager
[Service]
User=root
ExecStart=/opt/alertmanager-0.21.0.linux-amd64/alertmanager — config.file=/opt/alertmanager-0.21.0.linux-amd64/config.yml — web.external-url=http://192.168.111.2:9093/
[Install]
WantedBy=default.target
Set IP address according your IP VM through port 9093,
sudo systemctl daemon-reload
sudo systemctl enable alert_manager.service
sudo systemctl start alert_manager.service
Open your browser and access 192.168.111.2:9093
##4. Create Rules Alert
Make sure your has been installing Prometheus and change directory to your Prometheus,
cd /opt/prometheus-2.10.0.linux-amd64
Create rules and save, sudo vi node_rules.yml
groups:
- name: node.rules
rules:
- alert: InstanceDown
expr: up{job=”v-Sphere”} == 0
for: 1m
annotations:
summary: “Instance {{ $labels.instance }} down”
description: “Instance {{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minutes.”
Config about for instance down, according your Prometheus config, in the above config we use expr: up{job=”v-Sphere”} == 0
##5. Add node_rules.yml into Prometheus config that is config.yml
sudo vi config.yml and set config refer to picture below
##6. Restart Prometheus Server
sudo systemctl daemon-reload
sudo systemctl start prometheus_server.service
##7. Testing Alert Manager
# Shutdown node in vpshere job
# Access your IP prometheus:9090/targets
# Access your IP prometheus:9090/alerts, state node in vsphere job would be PENDING.
# Wait until FIRING.
# Access your IP Alert Manager:9093
# Check Slack Channel that we create earlier.
