Prometheus & Node Exporter Setup Documentation

Automated scripts to install and configure Prometheus and Node Exporter on Linux systems for easy monitoring and metric collection.

Introduction

Prometheus is a powerful open-source system monitoring and alerting toolkit that is widely used for gathering and storing metrics. In this setup, we will install and configure Prometheus along with Node Exporter, a Prometheus exporter that exposes system metrics such as CPU usage, memory usage, disk usage, and more. These metrics can then be collected and visualized in Grafana.

The repository provides automation scripts that simplify the installation and configuration process for Prometheus and Node Exporter, ensuring that they are set up as systemd services and configured correctly for monitoring on your Linux system.

Features

Installation Steps

Prerequisites

Prometheus Installation

Prometheus is an open-source monitoring and alerting toolkit that is used for collecting and querying metrics. This section will guide you through the automated Prometheus installation process using a script that will:

Step-by-Step Guide

Follow these instructions to install Prometheus using the provided script:

1. Download the Latest Version of Prometheus

The script first retrieves the latest release version of Prometheus from the GitHub API. It will download the appropriate version of Prometheus based on your system’s architecture (either linux-amd64 or linux-arm64).

Download URL Construction: The script constructs the download URL by fetching the latest release version and determining the correct architecture.

Latest Prometheus version: v2.38.1
Download URL: https://github.com/prometheus/prometheus/releases/download/v2.38.1/prometheus-2.38.1.linux-amd64.tar.gz

2. Architecture Detection

The script checks your system's architecture using the uname -m command. Based on the result, it will download the appropriate version of Prometheus:

3. Installation of Prometheus

The Prometheus binaries are downloaded and extracted into a temporary directory. Then, the script moves the Prometheus binaries and console files to their correct locations:

4. User and Directory Setup

The script creates a dedicated user for Prometheus and sets the proper permissions for the required directories:

The user prometheus is created with no home directory and no shell access, ensuring security.

5. Configuration Files

The script creates two essential configuration files for Prometheus:

6. Start Prometheus

Once the installation is complete, the script reloads systemd, enables the Prometheus service to start at boot, and then starts the service immediately.

If successful, Prometheus will be running on your system and listening on port 9090. To check the status of the Prometheus service, use:

sudo systemctl status prometheus

Running the Script

To run the installation script, simply execute the following command:

bash <(curl -s https://raw.githubusercontent.com/catalogfi/Prometheus-Node_Exporter-Setup/refs/heads/main/prometheus_setup.sh)

Troubleshooting

If Prometheus does not start correctly, check the system logs using the following command:

sudo journalctl -u prometheus.service --no-pager

For more detailed troubleshooting, review the /var/log/syslog and /var/log/messages files.

Accessing Prometheus

After installation, you can access Prometheus by navigating to the following URL in your web browser:

http://your-server-ip:9090

Log in using the admin username and the password you specified during setup.

Node Exporter Installation

Node Exporter is a Prometheus exporter that exposes hardware and OS metrics such as CPU, memory, disk, and network statistics. This guide walks you through the automated installation process for Node Exporter on your system using a bash script. The script does the following:

Step-by-Step Guide

Follow these steps to install Node Exporter using the provided bash script:

1. Architecture Detection

The script starts by detecting your system's architecture using the uname -m command. Based on the result, it determines which version of Node Exporter to download:

If your architecture is unsupported, the script will exit with an error message.

2. Install Required Dependencies

Depending on your Linux distribution, the script installs the necessary packages:

If the package manager is unsupported, the script prompts you to manually install the required packages.

3. Download Latest Node Exporter Release

The script uses GitHub’s API to fetch the latest release of Node Exporter and constructs the appropriate download URL based on your system’s architecture.

Download URL: https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

It then proceeds to download the tar.gz file from GitHub and verifies the file integrity before extracting it.

4. Installation and Setup

Once the download is complete, the script extracts the contents and moves the node_exporter binary to /usr/local/bin/ for easy access.

5. Creating a Dedicated User for Node Exporter

The script creates a dedicated, non-login user node_exporter with no shell access to ensure security. This minimizes potential attack vectors and ensures that Node Exporter runs with minimal privileges.

6. Enable and Start Node Exporter

Once the service is set up, the script reloads systemd, enables the Node Exporter service to start on boot, and starts the service immediately. The following commands are executed:


sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
    

8. Verify Installation

After installation, you can verify that Node Exporter is running correctly using the following command:

sudo systemctl status node_exporter

If everything is set up correctly, you should see output indicating that the service is active (running) on port 9100.

Running the Script

To execute the Node Exporter installation script, run the following command:

bash <(curl -s https://raw.githubusercontent.com/catalogfi/Prometheus-Node_Exporter-Setup/refs/heads/main/node_exporter_setup.sh)

Expected Output

After running the script, you should see output similar to the following:


Latest Node Exporter version: v0.18.1
Download URL: https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
Downloading Node Exporter version v0.18.1 for x86_64...
Node Exporter installed and running on port 9100.
    

Troubleshooting

If Node Exporter does not start correctly, you can check the logs for any error messages using the following command:

sudo journalctl -u node_exporter.service --no-pager

Additionally, make sure that the Node Exporter binary is executable and that the systemd service file is correctly configured.

Accessing Node Exporter

Node Exporter will be available on port 9100. To access the metrics, navigate to the following URL in your web browser:

http://your-server-ip:9100/metrics

Replace <your-server-ip> with the IP address of the machine where Node Exporter is installed.

Verify Installation

Check Prometheus Status

To ensure that Prometheus is running correctly, use the following command:

sudo systemctl status prometheus

If Prometheus is not running, check the logs for more information:

journalctl -u prometheus

Check Node Exporter Status

To ensure that Node Exporter is running correctly, use the following command:

sudo systemctl status node_exporter

If Node Exporter is not running, check the logs for more information:

journalctl -u node_exporter

Access Prometheus and Node Exporter

Once the installation is complete, you can access the following services:

Prometheus Web UI

Prometheus exposes a web-based dashboard for querying and viewing metrics. Access it via:

http://<your-server-ip>:9090

Node Exporter Metrics

Node Exporter exposes system metrics in a format that Prometheus can scrape. You can view the raw metrics by navigating to:

http://<your-server-ip>:9100/metrics

Setting Up Grafana Dashboard and Data Source

In this section, we will walk through the process of setting up Grafana and configuring the data source, so it can pull metrics from Prometheus. The setup involves using a script that will automatically configure a new data source in Grafana and create a pre-configured dashboard for visualizing the metrics collected from your Prometheus server.

Prerequisites

Step-by-Step Guide

The setup_dashboard_and_datasource.sh script will automate the following tasks:

The script asks for the following inputs:

Script Breakdown

The script will:

  1. Get system details: The script captures the system hostname, public IP address, and creates a unique data source name and UID based on the system’s information.
  2. Create a JSON payload: A JSON payload is generated that contains all the necessary configuration details, such as:
    • Data Source Name: Name for the Prometheus data source, such as hostname prometheus stream.
    • Data Source UID: A unique identifier for the data source (e.g., based on the fully qualified domain name).
    • Data Source URL: The Prometheus server URL, which typically includes the system's public IP address (e.g., http://:9090).
    • Dashboard Title: Title of the dashboard (e.g., hostname Instance Metrics Dashboard).
    • Dashboard Description: A description of what the dashboard represents, including a summary of the metrics it will show (CPU, memory, disk usage, etc.).
  3. Send the request: The script sends the JSON payload to the middleware server, which will forward the request to Grafana’s API to create the data source and dashboard.

Running the Script

To run the script, execute the following command in your terminal:

bash <(curl -s https://raw.githubusercontent.com/catalogfi/Prometheus-Node_Exporter-Setup/refs/heads/main/setup_dashboard_and_datasource.sh)

The script will prompt you for the following inputs:

Enter Grafana URL (e.g., http://localhost:3000):
Enter Grafana API Token:
Enter Middleware Server URL (e.g., http://middleware-server:5000):

Response and Confirmation

Once the request is processed, the script will output a response based on the success of the operation:

Accessing the Dashboard

Once the dashboard is created, you can access it by logging into Grafana and selecting the newly created dashboard. You should now see metrics for your system, including CPU usage, memory, disk usage, and network statistics.

Sample JSON Payload

Here’s an example of the JSON payload that the script generates:


{
  "data_source_name": "hostname prometheus stream",
  "data_source_uid": "hostname-unique-id",
  "data_source_url": "http://:9090",
  "dashboard_title": "hostname Instance Metrics Dashboard",
  "dashboard_description": "This dashboard monitors metrics for the hostname instance, providing real-time insights into CPU, memory, and network usage.",
  "basic_auth_user": "admin",
  "basic_auth_password": "password",
  "grafana_url": "http://localhost:3000",
  "grafana_token": "your-grafana-api-token"
}
    

Troubleshooting

Prometheus or Node Exporter is Not Starting

If either Prometheus or Node Exporter is not starting, you can check their respective service logs:

journalctl -u prometheus
journalctl -u node_exporter

Ensure that there are no conflicting processes or misconfigurations. If issues persist, consider restarting the services:

sudo systemctl restart prometheus
sudo systemctl restart node_exporter

Prometheus is Not Collecting Metrics

Ensure that Node Exporter is running correctly and that Prometheus is configured to scrape metrics from the correct target:

sudo systemctl status node_exporter

Check the Prometheus target configuration at http://<prometheus-server-ip>:9090/targets to ensure Node Exporter is listed and correctly scraped.