{"id":166,"date":"2026-08-17T00:55:51","date_gmt":"2026-08-17T00:55:51","guid":{"rendered":"http:\/\/127.0.0.1\/en\/2026\/08\/17\/monitoring-the-openstack-lab-prometheus-grafana-ceph\/"},"modified":"2026-08-17T00:55:51","modified_gmt":"2026-08-17T00:55:51","slug":"monitoring-the-openstack-lab-prometheus-grafana-ceph","status":"publish","type":"post","link":"https:\/\/wp-nv.radut.info\/ro\/2026\/08\/17\/monitoring-the-openstack-lab-prometheus-grafana-ceph\/","title":{"rendered":"Monitoring the OpenStack Lab: Prometheus, Grafana, Ceph"},"content":{"rendered":"

Scope:<\/strong> this covers how the OpenStack lab and its Ceph cluster are actually monitored \u2014 the exporter stack, what Ceph exposes natively, the Prometheus recording rules that make the dashboards fast, a tour of the real dashboard panels (with screenshots), day-2 Ceph status commands, and how to load-test the Ceph cluster itself with dd<\/code>\/fio<\/code>. For how the cloud and its network\/BGP are built, see OpenStack Lab Deployment<\/strong>; for what runs on top of it, see Kubernetes on OpenStack<\/strong>. This article doesn’t repeat either \u2014 it’s monitoring-only.<\/p>\n

1. The exporter stack<\/h2>\n

Every metric on these dashboards comes from one of these, laid down by a single Ansible playbook (monitoring.yml<\/code>) that targets the whole OpenStack host group plus one dedicated monitoring VM:<\/p>\n\n\n\n\n\n\n\n\n
Exporter<\/th>\nWhat it covers<\/th>\nWhere it runs<\/th>\n<\/tr>\n
node_exporter<\/code><\/td>\nStandard host metrics (CPU, memory, disk, network) for each physical OpenStack host<\/td>\nEvery OpenStack host, as a pinned-tag Docker container under a long-lived systemd unit<\/td>\n<\/tr>\n
cadvisor<\/td>\nContainer-level RAM\/CPU for every LXC container OSA creates (one per infra service) and any Ceph containers<\/td>\nEvery OpenStack host<\/td>\n<\/tr>\n
process-exporter<\/td>\nPer-process RAM\/CPU \u2014 what’s actually holding memory: ceph-osd<\/code>, mysqld<\/code>, rabbitmq<\/code>\/beam.smp<\/code>, nova-*<\/code>, ovs<\/code>, qemu<\/code> \u2014 beyond what Nova’s own VM-allocation view shows<\/td>\nEvery OpenStack host<\/td>\n<\/tr>\n
lvm-exporter (hansmi\/prometheus-lvm-exporter<\/code>)<\/td>\nLVM volume-group\/logical-volume usage \u2014 node_exporter has no native LVM collector, needed for both OpenStack block-storage VGs and Ceph OSD VGs<\/td>\nEvery OpenStack host<\/td>\n<\/tr>\n
openstack-exporter<\/td>\nCentralized Nova\/Neutron\/Cinder\/Glance\/Keystone API and resource metrics \u2014 queries Keystone once<\/em> for the whole cloud, not per-node<\/td>\nThe monitoring VM itself, not the OpenStack cluster (a separate inventory owns it)<\/td>\n<\/tr>\n
ceph-mgr Prometheus module<\/td>\nCeph cluster health, capacity, IOPS\/throughput, per-pool and (optionally) per-daemon perf counters<\/td>\nEvery Ceph mon\/mgr node, port 9283<\/strong><\/td>\n<\/tr>\n<\/table>\n

Two operational details worth knowing before you go looking for “missing” metrics:<\/p>\n

    \n
  • node_exporter’s image tag is pinned inside an existing systemd unit, never recreated.<\/strong> The unit was laid down years ago; bumping the version is a regex replace of the image tag in the unit file followed by a restart \u2014 nothing else about the host is touched. If you’re chasing a stale node_exporter build info, check the systemd unit file directly rather than assuming a fresh container spec.<\/li>\n
  • ceph-mgr’s Prometheus endpoint on standby mons returns HTTP 200 with an empty body<\/strong> \u2014 only the currently-active mgr returns a real metrics payload. That’s expected Ceph behavior, not a broken exporter; don’t page on it.<\/li>\n<\/ul>\n

    2. Ceph-side Prometheus configuration<\/h2>\n

    Beyond just enabling the module, a few explicit ceph config set<\/code> calls shape what the mgr actually exposes:<\/p>\n

    # Per-pool RBD stats only for the pools that matter (avoids scraping every pool in the cluster)\nceph config set mgr mgr\/prometheus\/rbd_stats_pools glance,cinder,nova,kuberbd\n\n# Per-daemon perf counters (OSD op latency, mon session counts, etc.) \u2014 off by default,\n# turned on deliberately since this cluster is small enough that the extra cardinality is fine\nceph config set mgr mgr\/prometheus\/exclude_perf_counters false\n\nceph mgr module enable dashboard\nceph mgr module enable devicehealth\nceph mgr module disable nfs   # not used, one less module to reason about\n\n# SMART-based device health, scraped periodically\nceph device scrape-health-metrics\nceph device check-health<\/code><\/pre>\n
    The dashboard module also gets a dedicated read-only user (separate from the admin<\/code> account) so the Ceph web dashboard can be shared without handing out cluster-admin.<\/p>\n
    3. Prometheus recording rules<\/h2>\n
    Recording rules exist for one reason here: pre-computing an expensive, high-cardinality join once<\/em> so every dashboard panel and alert downstream can query a cheap precomputed series instead of repeating the same join boilerplate. The real one worth understanding end-to-end is the per-VM CPU\/memory attribution chain:<\/p>\n
    # process-exporter groups every qemu guest process by groupname=\"qemu-vm-<instance-uuid>\".\n# This rule attaches the human project NAME to each OpenStack server by joining on tenant_id.\n- record: qemu_vm:server_info:by_uuid\n  expr: |\n    openstack_nova_server_status * on(tenant_id) group_left(project_name) (\n      label_replace(\n        label_replace(openstack_identity_project_info, \"tenant_id\", \"$1\", \"id\", \"(.*)\"),\n        \"project_name\", \"$1\", \"name\", \"(.*)\"\n      ) * 0 + 1\n    )\n\n# Real per-VM CPU, pulled out of the qemu-vm-<uuid> process group and joined to\n# name\/hypervisor\/project via the rule above.\n- record: qemu_vm:cpu_cores:by_uuid\n  expr: |\n    label_replace(\n      sum by (node, groupname) (rate(namedprocess_namegroup_cpu_seconds_total{groupname=~\"qemu-vm-.*\"}[5m])),\n      \"uuid\", \"$1\", \"groupname\", \"qemu-vm-(.*)\"\n    ) * on(uuid) group_left(name, hypervisor_hostname, project_name) (qemu_vm:server_info:by_uuid * 0 + 1)\n\n- record: qemu_vm:memory_resident_bytes:by_uuid\n  expr: |\n    label_replace(\n      sum by (node, groupname) (namedprocess_namegroup_memory_bytes{groupname=~\"qemu-vm-.*\", memtype=\"resident\"}),\n      \"uuid\", \"$1\", \"groupname\", \"qemu-vm-(.*)\"\n    ) * on(uuid) group_left(name, hypervisor_hostname, project_name) (qemu_vm:server_info:by_uuid * 0 + 1)\n\n# Same CPU figure, normalized to % of that VM's own hypervisor's real physical core count \u2014\n# lets you compare load across hosts with different core counts on one 0-100 scale.\n- record: qemu_vm:cpu_percent_of_host:by_uuid\n  expr: |\n    qemu_vm:cpu_cores:by_uuid * 100\n    \/ on(node) group_left() (count by (node) (count by (node, cpu) (node_cpu_seconds_total{app=\"openstack\",tier=\"host\"})))<\/code><\/pre>\n
    The mechanism this depends on: every qemu guest process carries -uuid <instance-uuid><\/code> in its own command line, and that UUID matches Nova’s own openstack_nova_server_status{uuid=...}<\/code> series exactly \u2014 so a process-exporter cmdline-regex capture is directly joinable to the real instance name\/project\/hypervisor with no separate lookup service needed.<\/p>\n
    A second, simpler recording rule (now<\/code>, expr: time()<\/code>) exists purely so alert-rule expressions can reference a stable “current time” series without repeating time()<\/code> inline in every rule.<\/p>\n
    4. Dashboard tour<\/h2>\n
    Two real, recent changes to the Hypervisor Overview dashboard show the actual iteration this monitoring stack goes through:<\/p>\n