Deploy Prometheus host monitoring complete body

Original link: https://blog.frytea.com/archives/664/

Full monitoring body = Prometheus + Node Exporter + cadvisor + grafana

Prometheus, as a rotating database, collects data from each server;
As a data collector, node-exporter accepts prometheus collection requests and reports various parameters of the host;
As a container data collector, vadvisor accepts prometheus collection requests and reports various parameters of the host docker;
Grafana acts as a kanban, querying data from prometheus for data visualization.

Deploy Promoetheus

Prepare the configuration file /data/docker/prometheus/prometheus.yml

 global: scrape_interval: 15s # By default, scrape targets every 15 seconds. # Attach these labels to any time series or alerts when communicating with # external systems (federation, remote storage, Alertmanager). external_labels: monitor: 'codelab-monitor' # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # Override the global default and scrape targets from this job every 5 seconds. scrape_interval: 5s static_configs: - targets: ['localhost:9090'] # 采集node exporter监控数据- job_name: 'vps2' static_configs: - targets: ['10.28.0.1:9100']

Prepare the data folder and set the access permission to nobody

 # 创建文件夹mkdir -p /data/docker/prometheus/data # 取得nobody的uid docker run --rm quay.io/prometheus/busybox cat /etc/passwd ··· nobody:x:65534:65534:nobody:/home:/bin/false # 配置文件夹权限chown 65534:65534 -R /data/docker/prometheus/data

start prometheus

 $ docker run -d \ --name=prometheus \ -p 9090:9090 \ --network=myDefault \ --restart always \ --volume /data/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \ --volume /data/docker/prometheus/data:/prometheus \ prom/prometheus:v2.40.4 \ --web.enable-lifecycle \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/prometheus \ --storage.tsdb.retention=365d $ docker run -d \ --name=prometheus \ -p 9090:9090 \ --network=myDefault \ --restart always \ --volume /data/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \ --volume /data/docker/prometheus/data:/prometheus \ prom/prometheus:v2.40.4 \ --web.enable-lifecycle \ --config.file=/etc/prometheus/prometheus.yml \ --storage.tsdb.path=/prometheus \ --storage.tsdb.retention=365d # --web.enable-lifecycle 允许热加载配置文件# 注意放在镜像名后面的内容为映射到内部的配置标志

Deploy Node Exporter to collect host data

It can be installed directly by compiling or using the package manager.

Compile and install

 # x86 $ curl -OL https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz $ tar -xzf node_exporter-1.5.0.linux-amd64.tar.gz $ cp node_exporter-1.5.0.linux-amd64/node_exporter /usr/local/bin/ # arm64 $ curl -OL https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-arm64.tar.gz $ tar -xzf node_exporter-1.5.0.linux-arm64.tar.gz $ cp node_exporter-1.5.0.linux-arm64/node_exporter /usr/local/bin/ $ sudo useradd -rs /bin/false nodeusr $ echo "[Unit] Description=Node Exporter After=network.target [Service] User=nodeusr Group=nodeusr Type=simple ExecStart=/usr/local/bin/node_exporter [Install] WantedBy=multi-user.target" > /etc/systemd/system/node_exporter.service $ sudo systemctl daemon-reload $ sudo systemctl enable node_exporter && systemctl start node_exporter

Ubuntu and other direct installation:

 $ apt search node_exporter Sorting... Done Full Text Search... Done prometheus-node-exporter-collectors/focal 0+git20200110.fc91c86-1 all Supplemental textfile collector scripts for Prometheus node_exporter $ apt install -y prometheus-node-exporter-collectors # 启动服务$ systemctl enable prometheus-node-exporter.service && systemctl start prometheus-node-exporter.service Synchronizing state of prometheus-node-exporter.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable prometheus-node-exporter # OpenWRT opkg update opkg install prometheus-node-exporter-lua \ prometheus-node-exporter-lua-nat_traffic \ prometheus-node-exporter-lua-netstat \ prometheus-node-exporter-lua-openwrt \ prometheus-node-exporter-lua-wifi \ prometheus-node-exporter-lua-wifi_stations curl localhost:9100/metrics cat /etc/config/prometheus-node-exporter-lua config prometheus-node-exporter-lua 'main' option listen_ipv6 '0' option listen_port '9100' option listen_interface 'lan' /etc/init.d/prometheus-node-exporter-lua restart

Reload prometheus configuration file

 # 方法一使用SIGHUP 信号kill -HUP $(pidof prometheus) # 方法二使用POST 请求，需--web.enable-lifecycle curl -X POST http://localhost:9090/-/reload # 实测两种方法在Docker 下均不生效，猜测外部修改必须重启才能映射入容器，原因待查

deploy cadvisor

 # 这里使用了我在docker hub 镜像的官方gcr 源镜像# 避免无法拉取问题# 普通docker hub 镜像即可拉取$ sudo docker run \ --restart always \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:ro \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --volume=/dev/disk/:/dev/disk:ro \ --publish=8080:8080 \ --detach=true \ --name=cadvisor \ --privileged \ --device=/dev/kmsg \ --net=myDefault \ songtianlun/cadvisor:v0.46.0 $ curl http://localhost:8080

deploy grafana

 docker run -d \ --name=grafana \ -p 3000:3000 \ --network=myDefault \ --restart always \ grafana/grafana

Default account: admin/admin

The Kanban template can be found here: https://grafana.com/grafana/dashboards/

references

This article is transferred from: https://blog.frytea.com/archives/664/
This site is only for collection, and the copyright belongs to the original author.