Skip to main content

Configure Node Monitoring

This guide explains how to set up Prometheus and Grafana to monitor your Kaia node.

1. Metric Configuration in Kaia​

Kaia provides the following flags for metric export:

  • --metric: Enables metric recording. This flag is typically used in conjunction with the --prometheus flag.
  • --prometheus: Enables exporting recorded metrics to a Prometheus server. This flag is typically used in conjunction with the --metric flag.
  • --prometheusport: Specifies the port for Prometheus metrics. Defaults to 61001.

To enable metrics and Prometheus exporting, set both METRICS and PROMETHEUS to 1 in your .conf file:


METRICS=1
PROMETHEUS=1

2. Setting up Prometheus​

Prometheus acts as the central system for monitoring, providing robust data mining capabilities to extract real-time data from your nodes and archive it.

Prometheus Hardware Requirements

Before setting up Prometheus, ensure your system meets the following hardware requirements:

  • Processor: At least 2 CPUs
  • Memory: Minimum of 4 GB RAM
  • Storage: At least 20 GB of free disk space

2.1 Installing Prometheus​

The following steps outline the manual installation process for Prometheus. Choose your operating system for specific instructions. For more information about Prometheus installation, refer to the official Prometheus documentation.

  1. Download the latest Prometheus release suitable for your architecture (e.g., darwin-amd64) from the official Prometheus download page. This guide uses version 2.53.3 as an example.

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.darwin-arm64.tar.gz


wget https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.linux-amd64.tar.gz

  1. Extract the downloaded archive and install binaries by moving them to /usr/local/bin/:

tar xvfz prometheus-2.53.3.darwin-arm64.tar.gz
mv prometheus-2.53.3.darwin-arm64/prometheus /usr/local/bin/
mv prometheus-2.53.3.darwin-arm64/promtool /usr/local/bin/


wget https://github.com/prometheus/prometheus/releases/download/v2.53.3/prometheus-2.53.3.linux-amd64.tar.gz
tar xvfz prometheus-2.53.3.linux-amd64.tar.gz
mv prometheus-2.53.3.linux-amd64/prometheus /usr/local/bin/
mv prometheus-2.53.3.linux-amd64/promtool /usr/local/bin/

  1. Remove the downloaded archive and extracted directory:

rm -rf prometheus-2.53.3.darwin-arm64.tar.gz prometheus-2.53.3.darwin-amd64


rm -rf prometheus-2.43.0.linux-amd64.tar.gz prometheus-2.43.0.linux-amd64

  1. Add Prometheus to your PATH environment variable to access Prometheus from any terminal session.

echo "export PATH=\"\$HOME/monitoring/prometheus:\$PATH\"" >> ~/.bashrc
source ~/.bashrc
# This assumes that a prometheus directory already exists in the HOME directory. If not, make a new one (mkdir -p $HOME/monitoring/prometheus).

2.2 Configuring Prometheus​

Prometheus needs to be configured to scrape metrics from your Kaia nodes.

:::info[Prometheus Configuration]

The prometheus.yml file configures Prometheus. The key sections are:

  • global: Sets global configuration parameters like evaluation_interval (how often Prometheus evaluates rules) and scrape_interval (how often Prometheus scrapes targets). 15 seconds is a reasonable starting point for both, but adjust based on your needs and block time.

  • scrape_configs: Defines the targets Prometheus monitors. The job_name identifies the target group. static_configs lists the target addresses. Replace <ip> with the IP address of your Kaia node and ensure the port (61001 by default) is correctly configured.

For more advanced configurations, refer to the Prometheus documentation.

:::

  1. Open the prometheus.yml file located at prometheus/prometheus.yml in a text editor.

  2. Ensure the scrape_configs section includes your Kaia nodes. Below is an example configuration:


global:
evaluation_interval: 15s
scrape_interval: 15s
scrape_configs:
- job_name: klaytn
static_configs:
- targets: #Replace `192.168.1.100` and `192.168.1.101` with the actual IP addresses of your Kaia nodes.
- "192.168.1.100:61001"
- "192.168.1.101:61001"
...

  1. Use promtool to check the configuration file for any syntax errors:

promtool check config prometheus/prometheus.yml

  1. Start Prometheus with your configuration file.

prometheus --config.file=prometheus/prometheus.yml

2.3 Setting up Prometheus Using Macro Script (macOS)​

This script automates the Prometheus installation and configuration process on macOS. Adapt it for other Prometheus versions and operating systems as needed.


rm -rf prometheus
echo "Installing Prometheus..."
curl -LO https://github.com/prometheus/prometheus/releases/download/v2.43.0/prometheus-2.43.0.darwin-arm64.tar.gz
tar xvfz prometheus-2.43.0.darwin-arm64.tar.gz > /dev/null 2>&1 && mv prometheus-2.43.0.darwin-arm64 prometheus && rm -rf prometheus-2.43.0.darwin-arm64 && rm -rf prometheus-2.43.0.darwin-arm64.tar.gz
echo "export PATH=\"$HOMEDIR/monitoring/prometheus:\$PATH\"" >> ~/.bashrc && source ~/.bashrc
# Generate Prometheus config file (prometheus.yml)
printf "%s\n" "global:" \
" evaluation_interval: 15s" \
" scrape_interval: 15s" \
"" \
"scrape_configs:" \
"- job_name: klaytn" \
" static_configs:" \
" - targets:" > prometheus/prometheus.yml
# Append target configurations for multiple nodes
for (( i=0; i<NUMOFNODE; i++ ))
do
# Replace <ip> and <port> with the actual IP address and port (61001) for each node
printf " - \"<ip>:%d\"\n" <port> >> prometheus/prometheus.yml
done

3. Setting up Grafana​

Grafana allows you to visualize the metrics collected by Prometheus through customizable dashboards.

Grafana System Requirements

Before setting up Grafana, ensure your system meets the minimum hardware and software requirements from the official Grafana documentation.

3.1 Installing Grafana​

Download and install Grafana using the appropriate method for your operating system. For example, you can install Grafana on macOS using Hombrew (brew install grafana). See the official Grafana installation guide for detailed instructions.

3.2 Configuring Grafana​

Set up Grafana to visualize the metrics collected by Prometheus.

  1. Start Grafana Server.

# macOS using Homebrew
brew services start grafana

For other operating systems, refer to the official Grafana documentation.

  1. Open a web browser and navigate to http://localhost:3000. Log in using the default credentials (admin/admin).

  2. Add Prometheus as a Data Source.

    • Navigate to Cofiguration -> Data Sources.
    • Click on Add data source.
    • Select Prometheus as the type.
    • Set the URL to http://localhost:9090 (modify if Prometheus is on a different server).
    • Click Save & Test to verify the connection.

  3. Add a Kaia dashboard and add a panel to visualize Kaia block number.

    • Create a new dashboard or navigate to an existing one.
    • Click Edit in the top-right corner, click Add in the dashboard header, and select Visualization in the drop-down to add a panel.
    • Under Query:
      1. Select your Prometheus as Data source.
      2. Enter klaytn_blockchain_head_blocknumber in the Metric field.
      3. In Options, select Custom from the Legend dropdown and enter {{instance}} as the custom legend format.
    • Click Apply to save the panel to your dashboard.
Additional Kaia Dashboards

For a complete pre-configured dashboard and automated provisioning setup, refer to the klaytn-deploy repository. This repository contains JSON files for pre-built dashboards and configuration files for provisioning data sources.

3.3 Setting up Grafana Using Macro Script (macOS)​

This script automates the Grafana installation process on macOS. Adapt it for other Grafana versions and operating systems as needed.


# Remove any existing Grafana installation
rm -rf grafana
# Install Grafana
echo "Installing Grafana..."
curl -O https://dl.grafana.com/enterprise/release/grafana-enterprise-8.4.5.darwin-arm64.tar.gz
tar -zxvf grafana-enterprise-8.4.5.darwin-arm64.tar.gz > /dev/null 2>&1 && mv grafana-8.4.5 grafana && rm -rf grafana-enterprise-8.4.5.darwin-arm64.tar.gz
echo "export PATH=\"$HOMEDIR/monitoring/grafana/bin:\$PATH\"" >> ~/.bashrc && source ~/.bashrc
# Generate Grafana dashboard config file
printf "%s\n" "apiVersion: 1" \
"providers:" \
"- name: 'klaytn'" \
" folder: ''" \
" options:" \
" path: conf/provisioning/dashboards" > grafana/conf/provisioning/dashboards/klaytn-dashboard.yml
# Generate Grafana datasource config file
printf "%s\n" "datasources:" \
"- is_default: true " \
" name: 'klaytn'" \
" type: 'prometheus'" \
" url: 'http://localhost:9090'" > grafana/conf/provisioning/datasources/klaytn.yml


# Clone the klaytn-deploy repository if not already cloned
if [ ! -d "klaytn-deploy" ]; then
echo "Cloning klaytn-deploy repository..."
git clone https://github.com/klaytn/klaytn-deploy.git
fi
# Copy Grafana configuration files from klaytn-deploy
cp klaytn-deploy/grafana/*.json grafana/conf/provisioning/dashboards/

4. Access Services​

After installation and configuration, access Prometheus and Grafana interfaces to verify that everything is set up correctly.

  • Prometheus Interface

    • URL: http://localhost:9090
    • Verification: Navigate to this URL in your browser. You should see the Prometheus web interface. Use the Graph tab to execute sample queries and ensure metrics are being scraped.

  • Grafana Interface

    • URL: http://localhost:3000
    • Default Credentials:
      • Username: admin
      • Password: admin
    • Verification: Upon first login, you'll be prompted to change the default password. After logging in, ensure that the Prometheus data source is correctly configured and that the Kaia dashboard displays metrics.
Make this page better