> For the complete documentation index, see [llms.txt](https://zeyad-abulaban.gitbook.io/notes/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://zeyad-abulaban.gitbook.io/notes/devops/monitoring.md).

# Monitoring

## Grafana Prometheus Setup

```bash
## Adding Repository
sudo apt install -y gnupg2 curl software-properties-common
curl -fsSL https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/grafana.gpg
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
## Installing grafana
sudo apt update
sudo apt -y install grafana
sudo systemctl enable --now grafana-server

## Check if everything is good
sudo systemctl status grafana-server.service

## Access Grafana on port 3000.
## Default logins:
## Username: admin
## Password: admin
```

### Prometheus Setup

* Prerequisites

```bash
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus
## sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done
```

Install Prometheus

```bash
sudo apt update
sudo apt -y install wget curl vim
mkdir -p /tmp/prometheus && cd /tmp/prometheus

## Download the latest binary file
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | wget -qi

## Extract the file
tar xvf prometheus*.tar.gz
cd prometheus*/
sudo mv prometheus promtool /usr/local/bin/

## Verify the installation
prometheus --version
promtool --version

## Moving configuration files
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
sudo mv consoles/ console_libraries/ /etc/prometheus/
cd "$HOME"
```

Create a Prometheus service file

```bash
sudo tee /etc/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=
SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF
```

Change directory permissions

```bash
for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R prometheus:prometheus /var/lib/prometheus/
```

Start the service

```bash
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
systemctl status prometheus
```

## Exporters

* [Nginx Exporter](https://github.com/nginxinc/nginx-prometheus-exporter)

```nginx
## In /etc/nginx/nginx.conf
server {
    listen 0.0.0.0:8080;
    location /stub_status {
        stub_status on;
    }
}
```

* [Node Exporter](https://devopscube.com/monitor-linux-servers-prometheus-node-exporter/)
* [BlackBox Exporter (probing)](https://devconnected.com/how-to-install-and-configure-blackbox-exporter-for-prometheus/)

## Alerting

For this one I will use [node-exporter alert rules](https://samber.github.io/awesome-prometheus-alerts/rules.html#host-and-hardware).

```bash
curl https://raw.githubusercontent.com/zAbuQasem/Misc/main/node-exporter-prom-alerts.yml -o alerts.yml
sudo cp alerts.yml /etc/prometheus/
## Verify with promtool
promtool check rules alerts.yml
```

Finally, load it with:

```yaml
global:
  scrape_interval: 15s

rule_files:
  - alerts.yml

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['172.17.0.1:9100']
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://zeyad-abulaban.gitbook.io/notes/devops/monitoring.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.