It's running as an app on the Truenas server. 30 days retention period.

WebUI: http://192.168.68.89:30104/query

🎉 Great Start! Prometheus is Running — Now Let's Feed It Data!

You're in a fantastic position! Think of Prometheus like an empty data warehouse — it's open for business, but it needs suppliers (called Exporters) to deliver goods (called Metrics). Right now, Prometheus is running but has nothing to scrape yet.

Let me walk you through exactly what to do, step by step.

🧠 First — Understand How Prometheus Gets Data

Before we touch anything, here's the mental model:

[Your App/System]  →  [Exporter]  →  [Prometheus SCRAPES it]  →  [Data Stored]

Prometheus is a "puller" — it doesn't receive data pushed to it (by default). Instead, it goes out on a schedule (every 15–60 seconds) and pulls metrics from targets called Exporters.

✅ Step 0 — Do This First: Verify Prometheus is Healthy

Before adding any data sources, confirm Prometheus itself is working correctly.

1. Open your WebUI: http://192.168.68.89:30104/

2. Click on Status → Targets

You should see at least one target: Prometheus scraping itself (prometheus job). It should say UP in green.

3. Run this test query in the query box:

up

If you see a result like up{job="prometheus"} = 1, you're healthy and ready to go! ✅

📋 Step 1 — Decide What You Want to Monitor

This is the most important planning step. Common things people monitor on a home server / TrueNAS setup:

For a first-time setup, I strongly recommend starting with Node Exporter — it gives you a goldmine of system-level metrics (CPU, memory, disk, network) with almost no configuration.

🔧 Step 2 — Deploy Node Exporter (on TrueNAS)

Since you're on TrueNAS with containers, deploy Node Exporter as another app/container.

Key settings when deploying Node Exporter:

• Image: prom/node-exporter:latest
• Port: 9100 (this is where it exposes metrics)
• Important mount (so it can see the host's real data):

volumes:
  - /proc:/host/proc:ro       # Read-only access to process info
  - /sys:/host/sys:ro         # Read-only access to system info
  - /:/rootfs:ro              # Read-only access to the filesystem

What are these mounts? Node Exporter needs to "peek" at the host operating system's internal files to read CPU, memory, and disk stats. The :ro means read-only — it can look but never touch. Totally safe. 🔒

The args to pass:

--path.procfs=/host/proc
--path.sysfs=/host/sys
--path.rootfs=/rootfs

Once deployed, verify it's working by visiting:

http://192.168.68.89:9100/metrics

You should see a wall of text — those are raw metrics! That's exactly what Prometheus will read. 🎉

⚙️ Step 3 — Tell Prometheus to Scrape Node Exporter

This is done via Prometheus's configuration file called prometheus.yml. This file is the brain of Prometheus — it tells it who to scrape and how often.

Here's what your config should look like after adding Node Exporter:

global:
  scrape_interval: 15s       # How often Prometheus pulls data from all targets
  evaluation_interval: 15s   # How often it evaluates alerting rules

scrape_configs:

  # Job 1: Prometheus scrapes itself (this is usually pre-configured)
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']  # Prometheus's own metrics endpoint

  # Job 2: Scrape Node Exporter for host system metrics
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['192.168.68.89:9100']  # IP of your TrueNAS host + Node Exporter port

📖 Line-by-Line Breakdown:

🔄 Step 4 — Reload Prometheus Config

After editing prometheus.yml, tell Prometheus to reload it without restarting:

# Option A: Send a reload signal via HTTP (if lifecycle API is enabled)
curl -X POST http://192.168.68.89:30104/-/reload

# Option B: Restart the Prometheus container from TrueNAS UI
# (Safe — 30-day retention data is stored in the volume, not lost on restart)

✅ Step 5 — Verify the New Target is Being Scraped

Go back to: http://192.168.68.89:30104/ → Status → Targets

You should now see two targets, both showing UP:

prometheus    http://localhost:9090/metrics       UP
node_exporter http://192.168.68.89:9100/metrics   UP

Then run this query to confirm data is flowing:

node_cpu_seconds_total

If you get results — you're ingesting data! 🎊

🗺️ Your Full Roadmap Summary

✅  DONE     → Prometheus installed & running
🔲  DO NEXT  → Deploy Node Exporter container (port 9100)
🔲  DO NEXT  → Edit prometheus.yml to add the scrape job
🔲  DO NEXT  → Reload config & verify targets are UP
⭐  AFTER    → Install Grafana & connect it to Prometheus
⭐  AFTER    → Import a pre-built dashboard (e.g., Node Exporter Full - Dashboard ID 1860)

💡 Pro Tips for Your Setup

• 📁 Find your prometheus.yml — On TrueNAS, it's usually in the app's persistent storage volume. Check under your app's settings for the config path.
• 🔒 30-day retention is great for a home server — that's plenty of history.
• 🎨 Grafana is your next big step — Prometheus stores the data, but Grafana makes it beautiful with dashboards and graphs. The two tools are almost always used together!

Would you like me to walk you through deploying Node Exporter on TrueNAS step-by-step, or jump ahead to setting up Grafana? 🚀