Prometheus & Node Exporter Setup Documentation

Introduction

Prometheus is a powerful open-source system monitoring and alerting toolkit that is widely used for gathering and storing metrics. In this setup, we will install and configure Prometheus along with Node Exporter, a Prometheus exporter that exposes system metrics such as CPU usage, memory usage, disk usage, and more. These metrics can then be collected and visualized in Grafana.

The repository provides automation scripts that simplify the installation and configuration process for Prometheus and Node Exporter, ensuring that they are set up as systemd services and configured correctly for monitoring on your Linux system.

Features

• Automated Prometheus Setup: Installs Prometheus, configures it, and sets it up as a systemd service.
• Automated Node Exporter Setup: Installs Node Exporter and sets it up as a systemd service for gathering system metrics.
• System Architecture Detection: Automatically detects your system's architecture (x86_64 or aarch64) and installs the appropriate version of Prometheus and Node Exporter.
• Dependency Management: Installs required dependencies such as curl, jq, wget, and more, based on your package manager (APT, YUM, or DNF).
• Systemd Services: Configures Prometheus and Node Exporter as systemd services to ensure they start on boot and can be managed via systemctl.

Installation Steps

Prerequisites

• A Linux-based server (Ubuntu, CentOS, Fedora, etc.)
• Root or sudo privileges to install packages and create system services.
• Internet access to download installation packages.

Prometheus Installation

Prometheus is an open-source monitoring and alerting toolkit that is used for collecting and querying metrics. This section will guide you through the automated Prometheus installation process using a script that will:

• Download the latest Prometheus version based on your system architecture (x86_64 or aarch64).
• Install Prometheus and necessary files.
• Create the necessary system directories and set up the proper permissions.
• Set up basic authentication for Prometheus web UI.
• Configure Prometheus to scrape metrics from the Node Exporter (installed on your machine).
• Create a systemd service to automatically start Prometheus on boot.

Step-by-Step Guide

Follow these instructions to install Prometheus using the provided script:

1. Download the Latest Version of Prometheus

The script first retrieves the latest release version of Prometheus from the GitHub API. It will download the appropriate version of Prometheus based on your system’s architecture (either linux-amd64 or linux-arm64).

Download URL Construction: The script constructs the download URL by fetching the latest release version and determining the correct architecture.

Latest Prometheus version: v2.38.1
Download URL: https://github.com/prometheus/prometheus/releases/download/v2.38.1/prometheus-2.38.1.linux-amd64.tar.gz

2. Architecture Detection

The script checks your system's architecture using the uname -m command. Based on the result, it will download the appropriate version of Prometheus:

• x86_64 (64-bit Intel/AMD) -> prometheus-linux-amd64
• aarch64 (ARM 64-bit) -> prometheus-linux-arm64

3. Installation of Prometheus

The Prometheus binaries are downloaded and extracted into a temporary directory. Then, the script moves the Prometheus binaries and console files to their correct locations:

• /usr/local/bin/ for Prometheus and Promtool binaries.
• /etc/prometheus/ for the console files and libraries.

4. User and Directory Setup

The script creates a dedicated user for Prometheus and sets the proper permissions for the required directories:

• /var/lib/prometheus - for storing Prometheus data.
• /etc/prometheus - for Prometheus configuration files.

The user prometheus is created with no home directory and no shell access, ensuring security.

5. Configuration Files

The script creates two essential configuration files for Prometheus:

• exporter.yml - Configures Prometheus to scrape metrics from the Node Exporter (installed on the same machine at port 9100).
• web.yml - Configures basic authentication for Prometheus, using an admin username and password (with password hash generated using htpasswd).

6. Start Prometheus

Once the installation is complete, the script reloads systemd, enables the Prometheus service to start at boot, and then starts the service immediately.

If successful, Prometheus will be running on your system and listening on port 9090. To check the status of the Prometheus service, use:

sudo systemctl status prometheus

Running the Script

To run the installation script, simply execute the following command:

bash <(curl -s https://raw.githubusercontent.com/catalogfi/Prometheus-Node_Exporter-Setup/refs/heads/main/prometheus_setup.sh)

Troubleshooting

If Prometheus does not start correctly, check the system logs using the following command:

sudo journalctl -u prometheus.service --no-pager

For more detailed troubleshooting, review the /var/log/syslog and /var/log/messages files.

Accessing Prometheus

After installation, you can access Prometheus by navigating to the following URL in your web browser:

http://your-server-ip:9090

Log in using the admin username and the password you specified during setup.

Node Exporter Installation

Node Exporter is a Prometheus exporter that exposes hardware and OS metrics such as CPU, memory, disk, and network statistics. This guide walks you through the automated installation process for Node Exporter on your system using a bash script. The script does the following:

• Detects your system's architecture (x86_64 or aarch64) and installs the appropriate version of Node Exporter.
• Checks for required dependencies and installs them (curl, jq, wget, and apache2-utils or httpd-tools depending on your system).
• Downloads the latest release of Node Exporter from GitHub.
• Installs Node Exporter, creates a dedicated user for security, and sets up the necessary system directories.
• Configures Node Exporter to run as a service using systemd and ensures it starts automatically on boot.

Step-by-Step Guide

Follow these steps to install Node Exporter using the provided bash script:

1. Architecture Detection

The script starts by detecting your system's architecture using the uname -m command. Based on the result, it determines which version of Node Exporter to download:

• x86_64 (Intel/AMD 64-bit) -> node_exporter-linux-amd64
• aarch64 (ARM 64-bit) -> node_exporter-linux-arm64

If your architecture is unsupported, the script will exit with an error message.

2. Install Required Dependencies

Depending on your Linux distribution, the script installs the necessary packages:

• Debian/Ubuntu: curl, jq, wget, and apache2-utils
• Red Hat/CentOS: curl, jq, wget, and httpd-tools
• Fedora: curl, jq, wget, and httpd-tools

If the package manager is unsupported, the script prompts you to manually install the required packages.

3. Download Latest Node Exporter Release

The script uses GitHub’s API to fetch the latest release of Node Exporter and constructs the appropriate download URL based on your system’s architecture.

Download URL: https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

It then proceeds to download the tar.gz file from GitHub and verifies the file integrity before extracting it.

4. Installation and Setup

Once the download is complete, the script extracts the contents and moves the node_exporter binary to /usr/local/bin/ for easy access.

5. Creating a Dedicated User for Node Exporter

The script creates a dedicated, non-login user node_exporter with no shell access to ensure security. This minimizes potential attack vectors and ensures that Node Exporter runs with minimal privileges.

6. Enable and Start Node Exporter

Once the service is set up, the script reloads systemd, enables the Node Exporter service to start on boot, and starts the service immediately. The following commands are executed:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
    

8. Verify Installation

After installation, you can verify that Node Exporter is running correctly using the following command:

sudo systemctl status node_exporter

If everything is set up correctly, you should see output indicating that the service is active (running) on port 9100.

Running the Script

To execute the Node Exporter installation script, run the following command:

bash <(curl -s https://raw.githubusercontent.com/catalogfi/Prometheus-Node_Exporter-Setup/refs/heads/main/node_exporter_setup.sh)

Expected Output

After running the script, you should see output similar to the following:

Latest Node Exporter version: v0.18.1
Download URL: https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
Downloading Node Exporter version v0.18.1 for x86_64...
Node Exporter installed and running on port 9100.
    

Troubleshooting

If Node Exporter does not start correctly, you can check the logs for any error messages using the following command:

sudo journalctl -u node_exporter.service --no-pager

Additionally, make sure that the Node Exporter binary is executable and that the systemd service file is correctly configured.

Accessing Node Exporter

Node Exporter will be available on port 9100. To access the metrics, navigate to the following URL in your web browser:

http://your-server-ip:9100/metrics

Replace <your-server-ip> with the IP address of the machine where Node Exporter is installed.

Verify Installation

Check Prometheus Status

To ensure that Prometheus is running correctly, use the following command:

sudo systemctl status prometheus

If Prometheus is not running, check the logs for more information:

journalctl -u prometheus

Check Node Exporter Status

To ensure that Node Exporter is running correctly, use the following command:

sudo systemctl status node_exporter

If Node Exporter is not running, check the logs for more information:

journalctl -u node_exporter

Access Prometheus and Node Exporter

Once the installation is complete, you can access the following services:

Prometheus Web UI

Prometheus exposes a web-based dashboard for querying and viewing metrics. Access it via:

http://<your-server-ip>:9090

Node Exporter Metrics

Node Exporter exposes system metrics in a format that Prometheus can scrape. You can view the raw metrics by navigating to:

http://<your-server-ip>:9100/metrics

Setting Up Grafana Dashboard and Data Source

In this section, we will walk through the process of setting up Grafana and configuring the data source, so it can pull metrics from Prometheus. The setup involves using a script that will automatically configure a new data source in Grafana and create a pre-configured dashboard for visualizing the metrics collected from your Prometheus server.

Prerequisites

• Grafana instance: Make sure that Grafana is installed and accessible on your system (e.g., http://your-server-ip:3000).
• Grafana API Token: You need a valid Grafana API token with permission to add data sources and dashboards. You can generate this token from the Grafana UI (under "Configuration" > "API Keys").
• Middleware Server: The script uses a middleware server to interact with the Grafana instance via API. Make sure that the middleware server is set up and running.

Step-by-Step Guide

The setup_dashboard_and_datasource.sh script will automate the following tasks:

• Configure Prometheus as a data source in Grafana.
• Create a dashboard in Grafana to visualize the metrics from the Prometheus instance.

The script asks for the following inputs:

• Grafana URL: The URL of your Grafana instance (e.g., http://localhost:3000).
• Grafana API Token: The token generated from your Grafana instance that allows API interaction.
• Middleware Server URL: The URL of the middleware server that will handle the requests to Grafana's API (e.g., http://middleware-server:5000).

Script Breakdown

The script will:

1. Get system details: The script captures the system hostname, public IP address, and creates a unique data source name and UID based on the system’s information.
2. Create a JSON payload: A JSON payload is generated that contains all the necessary configuration details, such as:
   • Data Source Name: Name for the Prometheus data source, such as hostname prometheus stream.
   • Data Source UID: A unique identifier for the data source (e.g., based on the fully qualified domain name).
   • Data Source URL: The Prometheus server URL, which typically includes the system's public IP address (e.g., http://:9090).
   • Dashboard Title: Title of the dashboard (e.g., hostname Instance Metrics Dashboard).
   • Dashboard Description: A description of what the dashboard represents, including a summary of the metrics it will show (CPU, memory, disk usage, etc.).
3. Send the request: The script sends the JSON payload to the middleware server, which will forward the request to Grafana’s API to create the data source and dashboard.

Running the Script

To run the script, execute the following command in your terminal:

bash <(curl -s https://raw.githubusercontent.com/catalogfi/Prometheus-Node_Exporter-Setup/refs/heads/main/setup_dashboard_and_datasource.sh)

The script will prompt you for the following inputs:

Enter Grafana URL (e.g., http://localhost:3000):

Enter Grafana API Token:

Enter Middleware Server URL (e.g., http://middleware-server:5000):

Response and Confirmation

Once the request is processed, the script will output a response based on the success of the operation:

• Success: If the dashboard and data source are created successfully, you will see a message: Dashboard created successfully!
• Failure: If there’s an error, you will receive a message detailing the issue, such as Failed to create dashboard. Please check the server logs.

Accessing the Dashboard

Once the dashboard is created, you can access it by logging into Grafana and selecting the newly created dashboard. You should now see metrics for your system, including CPU usage, memory, disk usage, and network statistics.

Sample JSON Payload

Here’s an example of the JSON payload that the script generates:

{
  "data_source_name": "hostname prometheus stream",
  "data_source_uid": "hostname-unique-id",
  "data_source_url": "http://:9090",
  "dashboard_title": "hostname Instance Metrics Dashboard",
  "dashboard_description": "This dashboard monitors metrics for the hostname instance, providing real-time insights into CPU, memory, and network usage.",
  "basic_auth_user": "admin",
  "basic_auth_password": "password",
  "grafana_url": "http://localhost:3000",
  "grafana_token": "your-grafana-api-token"
}
    

Troubleshooting

Prometheus or Node Exporter is Not Starting

If either Prometheus or Node Exporter is not starting, you can check their respective service logs:

journalctl -u prometheus

journalctl -u node_exporter

Ensure that there are no conflicting processes or misconfigurations. If issues persist, consider restarting the services:

sudo systemctl restart prometheus

sudo systemctl restart node_exporter

Prometheus is Not Collecting Metrics

Ensure that Node Exporter is running correctly and that Prometheus is configured to scrape metrics from the correct target:

sudo systemctl status node_exporter

Check the Prometheus target configuration at http://<prometheus-server-ip>:9090/targets to ensure Node Exporter is listed and correctly scraped.