Install Graphite and Grafana Ubuntu 16.04

Install Graphite and Grafana Ubuntu 16.04

In this tutorial I'm going to use MySQL as database for Graphite and Whisper files as database for the performance data.

Most people use PostgreSQL as database for Graphite, but I'm more familiar with MySQL…

As web server I'm using Apache. If you like to mess around with things, you can also use Nginx.

All commands needs to run as user root or via sudo.

Install Graphite

First of all, we need to install some dependencies

apt-get install graphite-web graphite-carbon mysql-server python-mysqldb  python-pymysql apache2 libapache2-mod-wsgi apt-transport-https ssl-cert

Create a MySQL User and Database for Graphite

As already mentioned, I'm using MySQL to store Graphite metadata.

CREATE USER 'graphite'@'localhost' IDENTIFIED BY 'password';
CREATE DATABASE IF NOT EXISTS `graphite` DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON `graphite`.* TO 'graphite'@'localhost';

Now edit the following values in the file /etc/graphite/local_settings.py.

The SECRET_KEY is use as salt, whenever graphite is going to hash something. Set something unique as value.

For example the output of the command uuidgen or so.

SECRET_KEY = 'UNSAFE_DEFAULT'

Set the Timezone of your monitoring nodes.

TIME_ZONE = 'Europe/Berlin'

In the last set, we need to configure the MySQL database connection

DATABASES = {
    'default': {
        'NAME': 'graphite',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'graphite',
        'PASSWORD': 'password',
        'HOST': '127.0.0.1',
        'PORT': '3306',
        'STORAGE_ENGINE': 'INNODB'
    }
}

Graphite comes with a tool, to create the database schema automatically. All you need to do is, run the following command:

graphite-manage syncdb

If you run into the error django.db.utils.IntegrityError: (1215, u'Cannot add foreign key constraint') just try again to run the command graphite-manage syncdb

Now open the file /etc/default/graphite-carbon and change the value to true to enable carbon cache on boot.

CARBON_CACHE_ENABLED=true

Configure Storage Time

Copy the following content to the file /etc/carbon/storage-schemas.conf to store all values prefixed with statusengine for 90 days.

[statusengine]
pattern = ^statusengine\.
retentions = 60:90d

If you change these values later, you need to delete the Whisper files. For more information please go to the Carbon documentation.

For example:

[statusengine]
pattern = ^statusengine\.
retentions = 60:90d

[carbon]
pattern = ^carbon\.
retentions = 60:90d

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

Now you can launch your Graphite instance.

systemctl start carbon-cache

Enable Graphite Web

a2dissite 000-default
cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available

Move Graphite Web to port 8888

In this case, we don't want to use the Graphite Web. We just need the TCP API and the HTTP API provided by Graphite.

Open the file /etc/apache2/sites-available/apache2-graphite.conf and change the port to 8888:

<VirtualHost *:8888>

Now add the port 8888 to the Apache Config file /etc/apache2/ports.conf:

Listen 80
Listen 8888

<IfModule ssl_module>
        Listen 443
</IfModule>

<IfModule mod_gnutls.c>
        Listen 443
</IfModule>
a2ensite apache2-graphite
systemctl restart apache2

Now you should able to browse to the Graphite Web via http://your-address:8888

 

Graphite Web on Port 8888 for Statusengine

 

Install Grafana

Before you start! Please take a look at the official documentation! http://docs.grafana.org/installation/debian/

echo "deb https://packagecloud.io/grafana/stable/debian/ jessie main" > /etc/apt/sources.list.d/grafana.list
curl https://packagecloud.io/gpg.key | apt-key add -
apt-get update
apt-get install grafana

systemctl daemon-reload
systemctl start grafana-server
systemctl enable grafana-server

Configure Apache as reverse proxy for Grafana

Create the file /etc/apache2/sites-available/apache2-grafana.conf with the following content:

<VirtualHost *:80>
    ServerName statusengine.org

    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteRule (.*) https://%{SERVER_NAME}/$1 [R,L]
</VirtualHost>

<VirtualHost *:443>
    ProxyPreserveHost On
    ProxyPass / http://127.0.0.1:3000/
    ProxyPassReverse / http://127.0.0.1:3000/

    SSLEngine On
    SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
    SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
</VirtualHost>

Notice: We added an automatically redirect from http to https using self-signed certificates.

Enable the new configuration

a2ensite apache2-grafana
a2enmod proxy
a2enmod proxy_http
a2enmod ssl
a2enmod xml2enc
a2enmod rewrite
systemctl restart apache2

Now you should able to browse to the Grafana Frontend via https://your-address.

Username: admin

Password: admin

 

Grafana via HTTPS

 

The Grafana configuration is located at /etc/grafana/grafana.ini. I would recommend you to take a look at this file.

Add Graphite as Datasource in Grafana

 

Add Graphite data source in Grafana

 

Configure Statusengine Worker to save Performance data to Graphite

Add your new Graphite instance to your Statusengine Worker Config in /opt/statusengine/worker/etc/config.yml

Statusengine Worker use the "plaintext protocol", an TCP connection to save metrics to Graphite.

Remember to change the IP Address!

process_perfdata: 1

number_perfdata_worker: 1

perfdata_backend:
# - crate
 - graphite
# - mysql

graphite_prefix: statusengine

graphite_address: 192.168.56.106

graphite_port: 2003

To enable the ne settings, restart Statusengine Worker

systemctl restart statusengine

Create your first Grafana Dashboard

Now you can create the first Graph in Grafana. Thanks to the Grafana UI this is super easy.

 

Create the first Graph in Grafana

 

If you have gaps in your Graphs, set Null value to connected

 

Grafana null value connected

 

Configure Statusengine Ui to load Performance data from Graphite

In the last step you need to enable Graphite as Performance Data Backend for Statusengine Ui. Open the file /usr/share/statusengine-ui/etc/config.yml and change the following values.

Remember to change the IP Address!

display_perfdata: 1

# Uncomment to enable
# CrateDB as Performance Data Backend
# CrateDB is the default at the moment
#perfdata_backend: crate

# Graphite as Performance Data Backend
perfdata_backend: graphite

# MySQL as Performance Data Backend
#perfdata_backend: mysql

graphite_prefix: statusengine

graphite_url: http://192.168.56.106:8888

graphite_use_basic_auth: 0

graphite_user: graphite

graphite_password: password

graphite_allow_self_signed_certificates: 0

 

Statusengine Ui Performance Data via Graphite Backend

 

Play around

I recommend you to play around with Grafana and get in touch if you are new to it.

This is an example Dashboard I created:

 

Grafana example dashboard dark

 

Delete old records from Whisper

Graphite uses Carbon/Whisper to store the performance data.

You want to delete old records, just rm the files from:

/var/lib/graphite/whisper

Read more

How to migrate Raspberry Pi 5 OS from micro SD to NVME m.2 SSD

首先我買了Raspberry Pi CM5後來買了Raspberry Pi CM5 I/O board來當個人電腦使用,系統是安裝在256GB SD卡上運行的很好。用久了在開啟較肥的程式像Web Browser或LiberOffice會有慢半拍的反應,而有了升級NVME m.2 SSD念頭。 因為Raspberry Pi 5支援的最快PCIe gen3 x 4就不去考慮快的Gen4 or Gen5 m.2 SSD。找了ADATA出的 LEGEND 710入門級的產品,會利用HMB(Host Memory Buffer)來加速I/O速度,因為是Raspberry Pi OS kernel會認不得而無法正常使用 事先在SD卡的/boot/firmware/cmdline.txt 加入 kernel command line參數如下,然後重開機m.

By Phillips Hsieh

How to document Home Lab and Network

運維機房和跨域的網路,會遇到各式需求與問題,用對工具才能分析問題,個人覺得最重要的是使用能處理問題的工具。 推薦目前想學和正在使用的平台與軟體,協助將公司/家用機房文件化 佈告欄任務管理 Focalboard 白板可管理任務指派 網路架構文件編寫 netbox 精細管理網路設備與連接線路 IP 資源管理 phpipam 專注網路IP分配 邏輯塊文件編寫 draw.io 視覺化概念圖 機房設備管理 ITDB 管理設備生命週期與使用者

By Phillips Hsieh

如何在Raspberry Pi4上安裝Proxmox for ARM64

第一步 準備好Raspberry Pi 4 / CM4 4GB RAM,這裡要留意CM4如果是買有內建eMMC storage會限制不能使用SD卡開機而限制本地空間容量,如果沒有NAS外接空間或使用USB開機的話,建議買CM4 Lite插上大容量SD卡 第二步 去Armbian官網下載最小化Debian bookworm image https://www.armbian.com/rpi4b/ Armbian 25.2.2 Bookworm Minimal / IOT 然後寫入SD/USB開機碟,寫入方法參考官方文件 https://github.com/raspberrypi/usbboot/blob/master/Readme.md Note: 官方提供的預先設定系統方法,可以在Armbian初次啟動自動化完成系統設定。連結在此 https://docs.armbian.com/User-Guide_Autoconfig/

By Phillips Hsieh

世界越快心越慢

在晚飯後的休息時間,我特別享受在客廳瀏灠youtube上各樣各式創作者的影音作品。很大不同於傳統媒體,節目多是針對大多數族群喜好挑選的,在youtube上我會依心情看無腦的動畫、一些旅拍記錄、新聞時事談論。 尤其在看了大量的Youtube的分享後,我真的感受到會限制我的是我的無知,特別是那些我想都沒想過的實際應用,在學習後大大幫助到我的生活和工作層面。 休息在家時,我喜歡想一些沒做過的菜,動手去設計生活和工作上的解決方案,自己是真的很難閒著沒事做。 如創作文章,陪養新的習慣都能感覺到成長的喜悅,是不同於吃喝玩樂的快樂的。 創作不去限制固定的形式,文字是創作、影像聲音也是創作,記錄生活也是創作,我想留下的就是創造—》實現—》回憶,這樣子的循環過程,在留下的足跡面看到自己一路上的成長、失敗、絕望、重新再來。 雖然大部份的時候去做這些創作也不明白有什麼特別的意義,但不去做也不會留下什麼,所以呀不如反事都去試試看,也許能有不一樣的水花也許有意想不到的結果,投資自己永遠不會是失敗的決定,不是嗎?先問問自己再開始計畫下一步,未來沒人說得準。 像最近看youtube仍大一群人在為DOS開

By Phillips Hsieh