Building a Grafana Dashboard for Partition Health

Partition problems are silent: creation stops, skew grows, rows accumulate in DEFAULT, retention stalls. None of them raise an error, and all of them are visible in five numbers. This guide builds the dashboard that shows those numbers, with the recording rules behind each panel and a layout that answers “is anything wrong?” before it answers “why?”. It puts the measurements from Partition Skew Detection & Monitoring on a screen, within Partition Monitoring & Failover Automation.

Prerequisites

Step 1 — Put the expensive work in recording rules

Panels should read pre-computed series, never compute them:

One expression, two consumers, one schedule A catalog view is scraped by the exporter into raw series. A recording rule aggregates those series every thirty seconds. The dashboard panel and the alert rule both read the recorded series, so they cannot disagree, and the expensive catalog query runs on the exporter's schedule rather than once per dashboard viewer. v_partition_healthcatalog query exporterscraped every 30 s recording ruleshard:partition_days_ahead:min dashboard panelsame expression alert rulesame expression, same threshold The catalog query runs on a fixed schedule regardless of how many people open the dashboard, and the panel cannot show green while the alert is firing, because both read the same series. Keep the rule file next to the dashboard JSON so a threshold change touches both in the same commit. 
# prometheus/rules/partitions.yml
groups:
  - name: partition-health
    interval: 30s
    rules:
      - record: shard:partition_days_ahead:min
        expr: min by (shard, table) (pg_partition_health_days_ahead)

      - record: shard:partition_skew_ratio:max
        expr: |
          max by (shard, parent) (pg_partition_size_bytes)
            / avg by (shard, parent) (pg_partition_size_bytes)

      - record: shard:partition_default_rows:sum
        expr: sum by (shard, parent) (pg_partition_default_rows)

      - record: shard:partition_pending_retire:sum
        expr: sum by (shard) (pg_partition_pending_retire)

Operational note: A thirty-second evaluation interval is plenty. These values change on the timescale of hours, and a shorter interval only multiplies the cost of the underlying catalog queries.

DBA tip: Name recording rules with the level:metric:operation convention. It makes a dashboard panel’s query self-documenting and prevents two people inventing two names for the same series.

Step 2 — Lay out the verdict first

Layout: verdict on the top row, explanation below The first row holds four single-value panels — minimum days ahead, maximum skew ratio, rows in default partitions and pending retirements — each coloured by threshold. Below them, three time-series panels show how those values have moved over the last week, a per-shard heatmap of partition sizes, and a table of the ten largest partitions. The top row answers whether anything is wrong; the rest explains why. days ahead (min)61 skew ratio (max)1.8 rows in DEFAULT0 pending retire1 days-ahead per shard, 7 days skew ratio per table, 7 days ten largest partitions — shard, parent, child, size, rows, last autovacuum alert < 21alert > 2.0 alert > 1000alert > 0 for 24 h explains a falling verdict explains a rising verdict the drill-down, for when the explanation is not enough Every number on the top row has a matching alert rule with the same threshold, so the dashboard and the pager never disagree.

Operational note: Use the same expression in the panel and in the alert rule — ideally the same recording rule. Dashboards and alerts computing “the same” value two ways is a reliable source of incident-time confusion.

DBA tip: Colour thresholds should match the alert thresholds exactly. A green panel next to a firing alert destroys trust in both.

Step 3 — Write the panels

{
  "title": "Days ahead (min across shards)",
  "type": "stat",
  "targets": [{ "expr": "min(shard:partition_days_ahead:min{shard=~\"$shard\"})" }],
  "fieldConfig": {
    "defaults": {
      "unit": "d",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "red",   "value": null },
          { "color": "orange","value": 21 },
          { "color": "green", "value": 45 }
        ]
      }
    }
  }
}
{
  "title": "Ten largest partitions",
  "type": "table",
  "targets": [{
    "expr": "topk(10, pg_partition_size_bytes{shard=~\"$shard\"})",
    "format": "table",
    "instant": true
  }],
  "transformations": [
    { "id": "organize", "options": { "excludeByName": { "job": true, "instance": true } } }
  ]
}
{
  "templating": {
    "list": [{
      "name": "shard",
      "type": "query",
      "query": "label_values(pg_partition_size_bytes, shard)",
      "includeAll": true,
      "multi": true,
      "current": { "text": "All", "value": "$__all" }
    }]
  }
}

Operational note: instant: true on the table panel is important. Without it Grafana renders a range query and the table shows one row per timestamp per partition, which is thousands of rows nobody wants.

SRE tip: Keep the dashboard JSON in version control and provision it from there. A hand-edited dashboard is lost the first time someone recreates the Grafana instance.

Step 4 — Add the panels that explain a change

The explanatory panel: which shard, and since when Seven shards hold a flat days-ahead value of about ninety while shard three declines steadily from day four, crossing the alert threshold on day eleven. The per-shard breakdown answers both questions an on-call engineer has — which shard, and when did it start — without any further query. 90 d60 d30 d0 alert threshold — 21 days shard_3 shards 0–2, 4–7 maintenance stopped here day 1day 8day 15 Breaking the series out by shard is the entire value of this panel. An aggregated "minimum across shards" line would show the same decline without saying which shard to look at, which is the first thing anyone needs to know. Keep the aggregate on the stat panel and the breakdown on the time series: verdict above, explanation below.

Operational note: Set the default time range to seven days. Partition problems develop over days, and a one-hour default hides exactly the trend the dashboard exists to show.

SRE tip: Add an annotation source for deploys and for maintenance job runs. Half of “when did this start” questions are answered by a vertical line on the chart.

Verification

Confirm the dashboard responds to a real change rather than only looking plausible:

# suspend maintenance on one shard in staging and watch the panel react
kubectl patch cronjob partition-maintenance-shard3 -p '{"spec":{"suspend":true}}'
# after a day, the recorded series should show the decline
shard:partition_days_ahead:min{shard="shard3"}
shard3  events   88
shard3  events   87
shard3  events   86

The stat panel should shift colour as the value crosses each threshold, and the alert rule using the same expression should fire at the same moment the panel turns red.

Keeping the Dashboard Honest Over Time

A dashboard degrades in a specific way: panels accumulate, thresholds drift from the alert rules, and the shard variable stops matching reality after a topology change. Three habits keep it useful for longer than a quarter.

Provision it from version control. Store the JSON alongside the alert rules that share its expressions, and apply both from the same pipeline. When a threshold changes, the diff shows the panel and the rule changing together, and a reviewer can see whether they still agree.

Delete panels rather than collapsing them. A collapsed row is a panel nobody reads and everybody preserves. If a panel has not informed a decision in six months, its query is still evaluated on every load and its presence still costs attention during an incident.

Re-derive the shard variable from the data, not from a list. label_values(pg_partition_size_bytes, shard) picks up a new shard the moment it starts reporting; a hard-coded list quietly omits it, and the omission is invisible precisely because the new shard is the one nobody is watching yet.

Finally, review the dashboard after every partitioning incident. The question worth asking is not “was the information there?” but “was it on the first screen?”. Every incident where the answer was buried in a drill-down is an argument for promoting one number to the verdict row — and, usually, for demoting another.

Failure mode table

Failure mode Root cause SRE mitigation
Dashboard is slow and adds load during incidents panels run catalog queries against the database directly on every refresh read Prometheus recording rules only; keep the expensive query on the exporter’s schedule
Panels and alerts disagree the alert rule and the panel compute the same idea with different expressions point both at the same recording rule and use identical thresholds
Nobody notices a degrading shard the overview aggregated across shards and hid a single outlier show the aggregate as the verdict and a per-shard breakdown below it

FAQ

How many panels should a partition dashboard have?

Five or six, arranged so the top row answers whether anything is wrong and the rest explain why. A dashboard with forty panels is a data browser, not a monitoring tool: during an incident nobody reads past the first screen, so the first screen has to carry the verdict.

Should the dashboard query the database directly?

No. Panels should read Prometheus recording rules fed by the exporter, so the expensive catalog queries run on a fixed schedule rather than once per viewer per refresh. A dashboard that runs catalog scans against production is itself a load source, and it fails exactly when the database is already struggling.

Does every shard need its own dashboard?

One dashboard with a shard variable is better than N dashboards. Set the variable to include an “All” option so the overview panels aggregate across the fleet, and let drill-down select a single shard. Copies per shard drift apart within a quarter and nobody updates all of them.

Back to Partition Skew Detection Monitoring