Dashboard Variables: One Dashboard, Many Services

Scenario: A platform team has thirty microservices and does not want to hand-build thirty nearly-identical dashboards - one per service - that all need updating every time a new panel is added to the standard layout.

New words, in plain English

• Dashboard variable - a placeholder in a dashboard that a viewer can change from a dropdown at the top of the screen, which then updates every panel's query at once.
• Templating - Grafana's word for building a dashboard around variables instead of hardcoded values, so one dashboard serves many cases.

Without variables, a panel's query has a real service name hardcoded directly into it - rate(http_requests_total{job="checkout-service"}[5m]) only ever shows checkout-service, forever, unless someone edits the panel. Copying that dashboard thirty times, one per service, means thirty separate things to maintain forever - a new panel added to the standard layout has to be manually copied into all thirty.

Dashboard variables solve this by replacing a hardcoded value with a placeholder, shown as a dropdown at the top of the dashboard. Every panel's query references the variable instead of a literal value, so changing the dropdown at the top instantly re-runs every panel's query with the new selection - one dashboard, reused for every service, module, or environment a viewer might want to look at.

A variable's list of choices can be typed in by hand for a small fixed set, but it is far more commonly populated dynamically with a PromQL query itself - for example, asking Prometheus "what are all the distinct values of the job label you currently know about?" so the dropdown always reflects whatever is actually being scraped, with no manual upkeep as services are added or removed.

Analogy: A hardcoded dashboard is a printed letter that says "Dear Checkout Service" - useful for exactly one recipient, and reprinting it thirty times for thirty recipients means thirty separate documents to keep in sync. A dashboard variable is a mail-merge template: "Dear {{service}}" - write it once, and a dropdown lets anyone pick which recipient's version they want to see, always pulling from the same up-to-date list of names.

A worked example

# Panel query WITHOUT a variable - locked to one service forever:
rate(http_requests_total{job="checkout-service"}[5m])

# Panel query WITH a variable - the same panel now works for any service
# the viewer picks from the dropdown at the top of the dashboard:
rate(http_requests_total{job="$service"}[5m])

# The variable's own definition - dynamically listing every distinct
# job Prometheus currently knows about, instead of a hand-typed list:
#   Name: service
#   Query: label_values(http_requests_total, job)

Variables compound well: a dashboard commonly has more than one, like $environment (production/staging) and $service layered together, letting one dashboard cover every service in every environment from a pair of dropdowns. This is the standard way large teams keep dashboard maintenance bounded as the number of services grows - the dashboard count stays flat while the underlying fleet grows.

Tip: Start simple: build one working dashboard for one real service with hardcoded values first, confirm the panels and queries are actually useful, and only then generalize it with variables. Templating a dashboard before you know it is useful just makes debugging harder.

Goal: Put this to work in the promgraf-import-default-dashboard lab. Open /labs/prometheus-grafana, pick promgraf-import-default-dashboard, and fix the real broken monitoring stack - a genuine Prometheus server scraping real targets, and a genuine Grafana instance querying it.