Step-by-Step Guide: Deploying Prometheus on Ubuntu for Real-World Monitoring

Many Prometheus guides barely scratch the surface—basic binaries, one scrape target, no context for real systems. Here’s a blueprint for installing Prometheus 2.44.0 on Ubuntu (20.04/22.04) with hardened defaults, service management, and system metrics via Node Exporter. This is actionable, production-minded setup, not just a toy install.

Why Use Prometheus?

Prometheus is the backbone of observability pipelines in cloud-native ops. Its time-series storage, PromQL-powered queries, and modular exporters are why it replaced Nagios in so many teams. Core strengths:

• Pull model via exporters (no complex agents)
• Efficient TSDB engine, native alerting, simple HA story
• Good fit for containerized and VM-heavy fleets

But without clean separation of privilege, configuration integrity, and a plan for exporter metrics, Prometheus can quickly become a liability.

Prerequisites

• Ubuntu 20.04 or 22.04, root or sudo access
• TCP/9090 open inbound for remote dashboard
• curl, tar, wget, and basic shell literacy

[1] Create Least-Privilege Prometheus User

Always run Prometheus as a non-root user. Set up a system account with no shell and no home:

sudo useradd --no-create-home --shell /bin/false prometheus

Ownership of all relevant files/directories must point to this user. Skipping this step undercuts OS-level protections.

[2] Install Prometheus Binaries (2.44.0)

Download and extract the latest stable release. Automate version pinning for improved reproducibility. Example:

cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.44.0/prometheus-2.44.0.linux-amd64.tar.gz
tar -xvf prometheus-2.44.0.linux-amd64.tar.gz
cd prometheus-2.44.0.linux-amd64

Copy binaries to system path:

sudo cp prometheus promtool /usr/local/bin/

Set up configuration and storage hierarchy:

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp -r consoles console_libraries /etc/prometheus/

Fix permissions:

sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

Note

Avoid putting the config and data under /opt unless you have a policy reason—upgrades and backups are easier with /etc and /var.

[3] Minimal Prometheus Configuration

Every Prometheus instance needs a job list. Start by watching itself. Save as /etc/prometheus/prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

• scrape_interval controls metric freshness vs. storage overhead. 15s is typical; sub-5s scrapes are rarely needed except for ultra-fast alerting.
• This initial config enables “self-monitoring”—critical for troubleshooting deadlocks or scrape failures.

[4] Register Prometheus as a systemd Service

Systemd enables reliable restarts and log capture. Create /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus Monitoring Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

Check the log:

sudo journalctl -u prometheus -f

Gotcha

Default TSDB storage (/var/lib/prometheus) writes can rapidly exhaust disk on high-cardinality workloads. No out-of-the-box auto-pruning. Always monitor free space.

[5] Access the Prometheus Web Interface

By default, Prometheus listens on port 9090. Log into http://<server-ip>:9090 and visit Status → Targets.

Try a live query:

up

Expected value:

• 1 indicates each target is up
• 0 means scrape failed

Errors here typically indicate firewall issues, misconfigured prometheus.yml, or systemd permissions.

[6] Extend Monitoring with Node Exporter

Without Node Exporter, you have no visibility into CPU, memory, or disk at the OS layer. The following covers direct binary install (v1.7.0), not Snap/apt, for version control and easy upgrades.

Fetch and install:

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

Add a non-privileged user:

sudo useradd --no-create-home --shell /bin/false node_exporter
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Systemd unit /etc/systemd/system/node_exporter.service:

[Unit]
Description=Node Exporter
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

Start and verify:

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

Add Node Exporter as a Prometheus Target

Append to scrape_configs in /etc/prometheus/prometheus.yml:

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

Restart Prometheus:

sudo systemctl restart prometheus

Node metrics (e.g. node_load1, node_memory_Active_bytes) should surface immediately.

Side Note

On multi-host deployments, node exporters are usually discovered via service discovery or file-based target lists, not hard-coded.

Operational Reminders

• Backups: Regularly ship /etc/prometheus, /var/lib/prometheus for disaster recovery. Yet—TSDB isn’t ideal for long-term cold backups due to potential index corruptions on restore. Consider remote_write for critical metrics.
• Upgrades: Prometheus and exporters should be upgraded independently. Read changelogs—backward-incompatible config options are rare but possible.
• Minor Tuning: For high ingest rates, increase open file limits and adjust --storage.tsdb.retention.time.
• Security: Out-of-the-box, Prometheus web UI is unauthenticated. Consider a reverse proxy (e.g., NGINX with mTLS) to restrict access.

What’s Next?

With Prometheus and Node Exporter running, add:

Also—inspect the effect of scrape_timeout vs. scrape_interval for exporters running on high-latency hosts. Metric lag is a common performance artifact.

Non-Obvious Tip

On busy hosts, node_exporter collectors (e.g. mountstats, filesystem) can cause high IO latency if not filtered. Use the --collector.filesystem.ignored-mount-points flag to skip bind mounts or ephemeral storage.

Prometheus isn’t perfect: scaling the TSDB to millions of metrics requires remote storage and sharding. Still, this setup is robust for most single-node or small-cluster install patterns.

For detailed patterns (multi-tenant, RBAC, or Service Discovery with Consul), layer on after validating the basics above. Always test metric queries and alert flow with synthetic failures before relying on this stack for NOC duty.

Happy monitoring.