linux monitoration - grafana and promethus

lufy

June 20, 2024

<div class="note-detail-wrapper">
<div class="note-detail note-detail-preview">
<div class="note-detail-markdown note-preview" data-markdown-root="true">
<h1>Overview</h1>
<p>grafana: for display to the web promethus: for data collection and provide data source to grafana node_exporter: for export mon data of linux os</p>
<p>So you need a server, for hosting your grafana and promethus. node_exporter should run on machines you want to monitor.</p>
<h1>How to:</h1>
<h2>Installation:</h2>
<p>dnf install -y golang-github-prometheus-node_exporter dnf install -y prometheus2 grafana</p>
<h2>Set up:</h2>
<h3>start services</h3>
<p>systemctl start prometheus systemctl start grafana-server systemctl start node_exporter</p>
<h3>Check services</h3>
<p>Open browser, test the followed urls(using the defautl ports): prometheus: http://:9090 node_exporter: http://:9100 . In some occation, node_exporter may use https rather than http. This may configure by editing <code>/etc/node_exporter/node_exporter.yaml</code>grafana: http://:3000</p>
<h3>Access</h3>
<p>grafana has default user and password: admin, admin node_exporter does set a user by default, except some settings already generated in <code>/etc/node_exporter/node_exporter.yaml</code>grafana doesn't set user by default.</p>
<h2>Access linux mon data (provided by node_exporter) using grafana web frontend</h2>
<h3>Adding data to promethus</h3>
<p>Add targets to prometheus configuration file <code>/etc/prometheus/prometheus.yml</code></p>
<pre><code class="hljs language-crmsh" data-highlighted="yes">vim /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: '<span class="hljs-keyword">node</span><span class="hljs-title">'
    static_configs</span>:
      - targets: [<span class="hljs-string">"10.7.62.89:9100"</span>,<span class="hljs-string">"10.7.62.96:9100"</span>]</code></pre>
<p>Then restart prometheus service by <code>systemctl restart prometheus</code>Now you prometheus could get mon data from node_exporter, you can check it at http://:9100, using a basic command <code>node_cpu_seconds_total</code>(using promQL) and visit http://:9100/targets to check targets status.</p>
<blockquote>
<p>PromQL You may find samples at <a href="https://prometheus.io/docs/prometheus/latest/querying/examples/" target="_blank" rel="external noopener noreferrer">samples</a> <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/" target="_blank" rel="external noopener noreferrer">Documentation</a></p>
</blockquote>
<h3>Add data source to grafana</h3>
<p>Open http://:3000 with browser, click cogwheel button -&gt; data source -&gt; new, select source type by click prometheus selection, input prometheus url (http://:9090 to the HTTP section, then save the data source.</p>
<h3>Add data dashboard to grafana</h3>
<p>Inside grafana web frontend, click dashboard -&gt; import, you can import a json formated dashboard to grafana (I feel it's to create a dash board on panel by another). You can find dashboards from <a href="https://grafana.com/grafana/dashboards/" target="_blank" rel="external noopener noreferrer">https://grafana.com/grafana/dashboards/</a>, in my instance, I imported <a href="https://grafana.com/grafana/dashboards/1860-node-exporter-full/" target="_blank" rel="external noopener noreferrer">Node Exporter Full</a>" by ID "1860"</p>
<p>Then all works done, you can view the dashboard from grafana-&gt;dashboard list.</p>
<h3>access data from promethus API</h3>
<p>The promethus API query URL is http://:9090/api/v1/query , the parameters should be query expression by promQL. For example: parameter:</p>
<pre><code class="hljs language-csp" data-highlighted="yes">{query: "100 - (avg by (instance) (rate(node_cpu_seconds_total{instance=<span class="hljs-string">'10.7.58.220:9100'</span>,mode=<span class="hljs-string">'idle'</span>}[120s]))*100)"
}</code></pre>
<p>Or using url with parameters:</p>
<p>http://:9090/api/v1/query?query=100%20-%20(avg%20by%20(instance)%20(rate(node_cpu_seconds_total{instance=%2710.7.58.220:9100%27,mode=%27idle%27}[120s]))*100)</p>
<p>This would git you a json object with cpu usage of node_exporter target named by "10.7.58.220:9100".</p>
<p>Response:</p>
<pre><code class="hljs language-prolog" data-highlighted="yes">{
    <span class="hljs-string">"status"</span>: <span class="hljs-string">"success"</span>,
    <span class="hljs-string">"data"</span>: {
        <span class="hljs-string">"resultType"</span>: <span class="hljs-string">"vector"</span>,
        <span class="hljs-string">"result"</span>: [
            {
                <span class="hljs-string">"metric"</span>: {
                    <span class="hljs-string">"instance"</span>: <span class="hljs-string">"10.7.62.97:9100"</span>
                },
                <span class="hljs-string">"value"</span>: [
                    <span class="hljs-number">1718854879.297</span>,
                    <span class="hljs-string">"3.455576053319959"</span>
                ]
            }
        ]
    }
}</code></pre>
<h3>Access grafana by anonymous user</h3>
<p>Need to change grafana configuration file</p>
<pre><code class="hljs language-makefile" data-highlighted="yes">vim /etc/grafana/grafana.ini
[auth.anonymous]
<span class="hljs-comment"># enable anonymous access</span>
enabled = true
org_name = uniontech
org_role = Viewer</code></pre>
<p>Then restart grafana-server.</p>
<blockquote>
<p>TIP: the org_name should be the same as organization that exists. You can change the name in admin page -&gt; preference -&gt; update organization name.</p>
</blockquote>
<h1>Issues</h1>
<p>In the panel, some data show error with like: Bad_data: 1:74: parse error: bad duration syntax: &ldquo;1m0&rdquo; error in grafana</p>
<p>This maybe the issue of grafana version. You can fix it by editing panel you imported, in dashboard setting, switch to JSON model, search <code>$__rate_interval</code>, replace with <code>$__interval</code>, and save and refresh. The resolution is sited at <a href="https://discuss.linuxcontainers.org/t/bad-data-1-parse-error-bad-duration-syntax-1m0-error-in-grafana/13350" target="_blank" rel="external noopener noreferrer">https://discuss.linuxcontainers.org/t/bad-data-1-parse-error-bad-duration-syntax-1m0-error-in-grafana/13350</a></p>
</div>
</div>
</div>

linux monitoration - grafana and promethus

Comments (1)

Leave a Comment