Configure Node Monitoring
This guide explains how to set up Prometheus and Grafana to monitor your Kaia node.
1. Metric Configuration in Kaia
Kaia provides the following flags for metric export:
--metric
: Enables metric recording. This flag is typically used in conjunction with the--prometheus
flag.--prometheus
: Enables exporting recorded metrics to a Prometheus server. This flag is typically used in conjunction with the--metric
flag.--prometheusport
: Specifies the port for Prometheus metrics. Defaults to61001
.
To enable metrics and Prometheus exporting, set both METRICS
and PROMETHEUS
to 1
in your .conf
file:
METRICS=1PROMETHEUS=1
2. Setting up Prometheus
Prometheus acts as the central system for monitoring, providing robust data mining capabilities to extract real-time data from your nodes and archive it.
Before setting up Prometheus, ensure your system meets the following hardware requirements:
- Processor: At least 2 CPUs
- Memory: Minimum of 4 GB RAM
- Storage: At least 20 GB of free disk space
2.1 Installing Prometheus
The following steps outline the manual installation process for Prometheus. Choose your operating system for specific instructions. For more information about Prometheus installation, refer to the official Prometheus documentation.
- Download the latest Prometheus release suitable for your architecture (e.g., darwin-amd64) from the official Prometheus download page. This guide uses version 2.53.3 as an example.
curl -LO https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.darwin-arm64.tar.gz
wget https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.linux-amd64.tar.gz
- Extract the downloaded archive and install binaries by moving them to
/usr/local/bin/
:
tar xvfz prometheus-2.53.3.darwin-arm64.tar.gzmv prometheus-2.53.3.darwin-arm64/prometheus /usr/local/bin/mv prometheus-2.53.3.darwin-arm64/promtool /usr/local/bin/
wget https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.linux-amd64.tar.gztar xvfz prometheus-2.53.3.linux-amd64.tar.gzmv prometheus-2.53.3.linux-amd64/prometheus /usr/local/bin/mv prometheus-2.53.3.linux-amd64/promtool /usr/local/bin/
- Remove the downloaded archive and extracted directory:
rm -rf prometheus-2.53.3.darwin-arm64.tar.gz prometheus-2.53.3.darwin-amd64
rm -rf prometheus-2.43.0.linux-amd64.tar.gz prometheus-2.43.0.linux-amd64
- Add Prometheus to your
PATH
environment variable to access Prometheus from any terminal session.
echo "export PATH=\"\$HOME/monitoring/prometheus:\$PATH\"" >> ~/.bashrcsource ~/.bashrc# This assumes that a prometheus directory already exists in the HOME directory. If not, make a new one (mkdir -p $HOME/monitoring/prometheus).
2.2 Configuring Prometheus
Prometheus needs to be configured to scrape metrics from your Kaia nodes.
:::info[Prometheus Configuration]
The prometheus.yml
file configures Prometheus. The key sections are:
-
global
: Sets global configuration parameters likeevaluation_interval
(how often Prometheus evaluates rules) andscrape_interval
(how often Prometheus scrapes targets). 15 seconds is a reasonable starting point for both, but adjust based on your needs and block time. -
scrape_configs
: Defines the targets Prometheus monitors. Thejob_name
identifies the target group.static_configs
lists the target addresses. Replace<ip>
with the IP address of your Kaia node and ensure the port (61001
by default) is correctly configured.
For more advanced configurations, refer to the Prometheus documentation.
:::
-
Open the
prometheus.yml
file located atprometheus/prometheus.yml
in a text editor. -
Ensure the
scrape_configs
section includes your Kaia nodes. Below is an example configuration:
global: evaluation_interval: 15s scrape_interval: 15sscrape_configs:- job_name: klaytn static_configs: - targets: #Replace `192.168.1.100` and `192.168.1.101` with the actual IP addresses of your Kaia nodes. - "192.168.1.100:61001" - "192.168.1.101:61001" ...
- Use
promtool
to check the configuration file for any syntax errors:
promtool check config prometheus/prometheus.yml
- Start Prometheus with your configuration file.
prometheus --config.file=prometheus/prometheus.yml
2.3 Setting up Prometheus Using Macro Script (macOS)
This script automates the Prometheus installation and configuration process on macOS. Adapt it for other Prometheus versions and operating systems as needed.
rm -rf prometheusecho "Installing Prometheus..."curl -LO https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.darwin-arm64.tar.gztar xvfz prometheus-2.43.0.darwin-arm64.tar.gz > /dev/null 2>&1 && mv prometheus-2.43.0.darwin-arm64 prometheus && rm -rf prometheus-2.43.0.darwin-arm64 && rm -rf prometheus-2.43.0.darwin-arm64.tar.gzecho "export PATH=\"$HOMEDIR/monitoring/prometheus:\$PATH\"" >> ~/.bashrc && source ~/.bashrc# Generate Prometheus config file (prometheus.yml)printf "%s\n" "global:" \ " evaluation_interval: 15s" \ " scrape_interval: 15s" \ "" \ "scrape_configs:" \ "- job_name: klaytn" \ " static_configs:" \ " - targets:" > prometheus/prometheus.yml# Append target configurations for multiple nodesfor (( i=0; i<NUMOFNODE; i++ ))do # Replace <ip> and <port> with the actual IP address and port (61001) for each node printf " - \"<ip>:%d\"\n" <port> >> prometheus/prometheus.ymldone
3. Setting up Grafana
Grafana allows you to visualize the metrics collected by Prometheus through customizable dashboards.
Before setting up Grafana, ensure your system meets the minimum hardware and software requirements from the official Grafana documentation.
3.1 Installing Grafana
Download and install Grafana using the appropriate method for your operating system. For example, you can install Grafana on macOS using Hombrew (brew install grafana
). See the official Grafana installation guide for detailed instructions.
3.2 Configuring Grafana
Set up Grafana to visualize the metrics collected by Prometheus.
- Start Grafana Server.
# macOS using Homebrewbrew services start grafana
For other operating systems, refer to the official Grafana documentation.
-
Open a web browser and navigate to
http://localhost:3000
. Log in using the default credentials (admin/admin). -
Add Prometheus as a Data Source.
- Navigate to Cofiguration -> Data Sources.
- Click on Add data source.
- Select Prometheus as the type.
- Set the URL to
http://localhost:9090
(modify if Prometheus is on a different server). - Click Save & Test to verify the connection.
-
Add a Kaia dashboard and add a panel to visualize Kaia block number.
- Create a new dashboard or navigate to an existing one.
- Click Edit in the top-right corner, click Add in the dashboard header, and select Visualization in the drop-down to add a panel.
- Under Query:
- Select your Prometheus as Data source.
- Enter
klaytn_blockchain_head_blocknumber
in the Metric field. - In Options, select Custom from the Legend dropdown and enter
{{instance}}
as the custom legend format.
- Click Apply to save the panel to your dashboard.
For a complete pre-configured dashboard and automated provisioning setup, refer to the klaytn-deploy repository. This repository contains JSON files for pre-built dashboards and configuration files for provisioning data sources.
3.3 Setting up Grafana Using Macro Script (macOS)
This script automates the Grafana installation process on macOS. Adapt it for other Grafana versions and operating systems as needed.
# Remove any existing Grafana installationrm -rf grafana# Install Grafanaecho "Installing Grafana..."curl -O https://dl.grafana.com/enterprise/release/grafana-enterprise-8.4.5.darwin-arm64.tar.gztar -zxvf grafana-enterprise-8.4.5.darwin-arm64.tar.gz > /dev/null 2>&1 && mv grafana-8.4.5 grafana && rm -rf grafana-enterprise-8.4.5.darwin-arm64.tar.gzecho "export PATH=\"$HOMEDIR/monitoring/grafana/bin:\$PATH\"" >> ~/.bashrc && source ~/.bashrc# Generate Grafana dashboard config fileprintf "%s\n" "apiVersion: 1" \ "providers:" \ "- name: 'klaytn'" \ " folder: ''" \ " options:" \ " path: conf/provisioning/dashboards" > grafana/conf/provisioning/dashboards/klaytn-dashboard.yml# Generate Grafana datasource config fileprintf "%s\n" "datasources:" \ "- is_default: true " \ " name: 'klaytn'" \ " type: 'prometheus'" \ " url: 'http://localhost:9090'" > grafana/conf/provisioning/datasources/klaytn.yml
# Clone the klaytn-deploy repository if not already clonedif [ ! -d "klaytn-deploy" ]; then echo "Cloning klaytn-deploy repository..." git clone https://github.com/klaytn/klaytn-deploy.gitfi# Copy Grafana configuration files from klaytn-deploycp klaytn-deploy/grafana/*.json grafana/conf/provisioning/dashboards/
4. Access Services
After installation and configuration, access Prometheus and Grafana interfaces to verify that everything is set up correctly.
-
Prometheus Interface
- URL:
http://localhost:9090
- Verification: Navigate to this URL in your browser. You should see the Prometheus web interface. Use the Graph tab to execute sample queries and ensure metrics are being scraped.
- URL:
-
Grafana Interface
- URL:
http://localhost:3000
- Default Credentials:
- Username:
admin
- Password:
admin
- Username:
- Verification: Upon first login, you'll be prompted to change the default password. After logging in, ensure that the Prometheus data source is correctly configured and that the Kaia dashboard displays metrics.
- URL: