Overview

grafana: for display to the web promethus: for data collection and provide data source to grafana node_exporter: for export mon data of linux os

So you need a server, for hosting your grafana and promethus. node_exporter should run on machines you want to monitor.

How to:

Installation:

dnf install -y golang-github-prometheus-node_exporter dnf install -y prometheus2 grafana

Set up:

start services

systemctl start prometheus systemctl start grafana-server systemctl start node_exporter

Check services

Open browser, test the followed urls(using the defautl ports): prometheus: http://:9090 node_exporter: http://:9100 . In some occation, node_exporter may use https rather than http. This may configure by editing /etc/node_exporter/node_exporter.yamlgrafana: http://:3000

Access

grafana has default user and password: admin, admin node_exporter does set a user by default, except some settings already generated in /etc/node_exporter/node_exporter.yamlgrafana doesn't set user by default.

Access linux mon data (provided by node_exporter) using grafana web frontend

Adding data to promethus

Add targets to prometheus configuration file /etc/prometheus/prometheus.yml

vim /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ["10.7.62.89:9100","10.7.62.96:9100"]

Then restart prometheus service by systemctl restart prometheusNow you prometheus could get mon data from node_exporter, you can check it at http://:9100, using a basic command node_cpu_seconds_total(using promQL) and visit http://:9100/targets to check targets status.

PromQL You may find samples at samples Documentation

Add data source to grafana

Open http://:3000 with browser, click cogwheel button -> data source -> new, select source type by click prometheus selection, input prometheus url (http://:9090 to the HTTP section, then save the data source.

Add data dashboard to grafana

Inside grafana web frontend, click dashboard -> import, you can import a json formated dashboard to grafana (I feel it's to create a dash board on panel by another). You can find dashboards from https://grafana.com/grafana/dashboards/, in my instance, I imported Node Exporter Full" by ID "1860"

Then all works done, you can view the dashboard from grafana->dashboard list.

access data from promethus API

The promethus API query URL is http://:9090/api/v1/query , the parameters should be query expression by promQL. For example: parameter:

{query: "100 - (avg by (instance) (rate(node_cpu_seconds_total{instance='10.7.58.220:9100',mode='idle'}[120s]))*100)"
}

Or using url with parameters:

http://:9090/api/v1/query?query=100%20-%20(avg%20by%20(instance)%20(rate(node_cpu_seconds_total{instance=%2710.7.58.220:9100%27,mode=%27idle%27}[120s]))*100)

This would git you a json object with cpu usage of node_exporter target named by "10.7.58.220:9100".

Response:

{
    "status": "success",
    "data": {
        "resultType": "vector",
        "result": [
            {
                "metric": {
                    "instance": "10.7.62.97:9100"
                },
                "value": [
                    1718854879.297,
                    "3.455576053319959"
                ]
            }
        ]
    }
}

Access grafana by anonymous user

Need to change grafana configuration file

vim /etc/grafana/grafana.ini
[auth.anonymous]
# enable anonymous access
enabled = true
org_name = uniontech
org_role = Viewer

Then restart grafana-server.

TIP: the org_name should be the same as organization that exists. You can change the name in admin page -> preference -> update organization name.

Issues

In the panel, some data show error with like: Bad_data: 1:74: parse error: bad duration syntax: “1m0” error in grafana

This maybe the issue of grafana version. You can fix it by editing panel you imported, in dashboard setting, switch to JSON model, search $__rate_interval, replace with $__interval, and save and refresh. The resolution is sited at https://discuss.linuxcontainers.org/t/bad-data-1-parse-error-bad-duration-syntax-1m0-error-in-grafana/13350