linux monitoration - grafana and promethus lufy June 20, 2024 <div class="note-detail-wrapper"> <div class="note-detail note-detail-preview"> <div class="note-detail-markdown note-preview" data-markdown-root="true"> <h1>Overview</h1> <p>grafana: for display to the web promethus: for data collection and provide data source to grafana node_exporter: for export mon data of linux os</p> <p>So you need a server, for hosting your grafana and promethus. node_exporter should run on machines you want to monitor.</p> <h1>How to:</h1> <h2>Installation:</h2> <p>dnf install -y golang-github-prometheus-node_exporter dnf install -y prometheus2 grafana</p> <h2>Set up:</h2> <h3>start services</h3> <p>systemctl start prometheus systemctl start grafana-server systemctl start node_exporter</p> <h3>Check services</h3> <p>Open browser, test the followed urls(using the defautl ports): prometheus: http://:9090 node_exporter: http://:9100 . In some occation, node_exporter may use https rather than http. This may configure by editing <code>/etc/node_exporter/node_exporter.yaml</code>grafana: http://:3000</p> <h3>Access</h3> <p>grafana has default user and password: admin, admin node_exporter does set a user by default, except some settings already generated in <code>/etc/node_exporter/node_exporter.yaml</code>grafana doesn't set user by default.</p> <h2>Access linux mon data (provided by node_exporter) using grafana web frontend</h2> <h3>Adding data to promethus</h3> <p>Add targets to prometheus configuration file <code>/etc/prometheus/prometheus.yml</code></p> <pre><code class="hljs language-crmsh" data-highlighted="yes">vim /etc/prometheus/prometheus.yml scrape_configs: - job_name: '<span class="hljs-keyword">node</span><span class="hljs-title">' static_configs</span>: - targets: [<span class="hljs-string">"10.7.62.89:9100"</span>,<span class="hljs-string">"10.7.62.96:9100"</span>]</code></pre> <p>Then restart prometheus service by <code>systemctl restart prometheus</code>Now you prometheus could get mon data from node_exporter, you can check it at http://:9100, using a basic command <code>node_cpu_seconds_total</code>(using promQL) and visit http://:9100/targets to check targets status.</p> <blockquote> <p>PromQL You may find samples at <a href="https://prometheus.io/docs/prometheus/latest/querying/examples/" target="_blank" rel="external noopener noreferrer">samples</a> <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/" target="_blank" rel="external noopener noreferrer">Documentation</a></p> </blockquote> <h3>Add data source to grafana</h3> <p>Open http://:3000 with browser, click cogwheel button -> data source -> new, select source type by click prometheus selection, input prometheus url (http://:9090 to the HTTP section, then save the data source.</p> <h3>Add data dashboard to grafana</h3> <p>Inside grafana web frontend, click dashboard -> import, you can import a json formated dashboard to grafana (I feel it's to create a dash board on panel by another). You can find dashboards from <a href="https://grafana.com/grafana/dashboards/" target="_blank" rel="external noopener noreferrer">https://grafana.com/grafana/dashboards/</a>, in my instance, I imported <a href="https://grafana.com/grafana/dashboards/1860-node-exporter-full/" target="_blank" rel="external noopener noreferrer">Node Exporter Full</a>" by ID "1860"</p> <p>Then all works done, you can view the dashboard from grafana->dashboard list.</p> <h3>access data from promethus API</h3> <p>The promethus API query URL is http://:9090/api/v1/query , the parameters should be query expression by promQL. For example: parameter:</p> <pre><code class="hljs language-csp" data-highlighted="yes">{query: "100 - (avg by (instance) (rate(node_cpu_seconds_total{instance=<span class="hljs-string">'10.7.58.220:9100'</span>,mode=<span class="hljs-string">'idle'</span>}[120s]))*100)" }</code></pre> <p>Or using url with parameters:</p> <p>http://:9090/api/v1/query?query=100%20-%20(avg%20by%20(instance)%20(rate(node_cpu_seconds_total{instance=%2710.7.58.220:9100%27,mode=%27idle%27}[120s]))*100)</p> <p>This would git you a json object with cpu usage of node_exporter target named by "10.7.58.220:9100".</p> <p>Response:</p> <pre><code class="hljs language-prolog" data-highlighted="yes">{ <span class="hljs-string">"status"</span>: <span class="hljs-string">"success"</span>, <span class="hljs-string">"data"</span>: { <span class="hljs-string">"resultType"</span>: <span class="hljs-string">"vector"</span>, <span class="hljs-string">"result"</span>: [ { <span class="hljs-string">"metric"</span>: { <span class="hljs-string">"instance"</span>: <span class="hljs-string">"10.7.62.97:9100"</span> }, <span class="hljs-string">"value"</span>: [ <span class="hljs-number">1718854879.297</span>, <span class="hljs-string">"3.455576053319959"</span> ] } ] } }</code></pre> <h3>Access grafana by anonymous user</h3> <p>Need to change grafana configuration file</p> <pre><code class="hljs language-makefile" data-highlighted="yes">vim /etc/grafana/grafana.ini [auth.anonymous] <span class="hljs-comment"># enable anonymous access</span> enabled = true org_name = uniontech org_role = Viewer</code></pre> <p>Then restart grafana-server.</p> <blockquote> <p>TIP: the org_name should be the same as organization that exists. You can change the name in admin page -> preference -> update organization name.</p> </blockquote> <h1>Issues</h1> <p>In the panel, some data show error with like: Bad_data: 1:74: parse error: bad duration syntax: “1m0” error in grafana</p> <p>This maybe the issue of grafana version. You can fix it by editing panel you imported, in dashboard setting, switch to JSON model, search <code>$__rate_interval</code>, replace with <code>$__interval</code>, and save and refresh. The resolution is sited at <a href="https://discuss.linuxcontainers.org/t/bad-data-1-parse-error-bad-duration-syntax-1m0-error-in-grafana/13350" target="_blank" rel="external noopener noreferrer">https://discuss.linuxcontainers.org/t/bad-data-1-parse-error-bad-duration-syntax-1m0-error-in-grafana/13350</a></p> </div> </div> </div>
Comments (1)
Leave a Comment
uomfvkjnvrpqixundfernzeuwqkztx