Dynamic Inventory in Production

Scenario: The static inventory was last accurate in March. Since then the autoscaler has replaced every web server twice, and a decommissioned database host is still listed - so --limit dbservers targets a machine that no longer exists.

New words, in plain English

• Inventory plugin - the modern mechanism: a YAML config Ansible executes to build hosts and groups from a live source.
• keyed_groups - automatically create groups from a tag or attribute on each discovered host.
• compose - build host variables from the discovered data.
• Inventory caching - storing the generated inventory so a source outage does not stop everything.
• Source of truth - the system that authoritatively knows what exists.

Static inventories rot. The question in production is not whether to use dynamic inventory but which source of truth to trust and what to do when it is unavailable.

An inventory plugin is a small YAML file naming a plugin and its options. Cloud providers, container platforms and asset databases ship them. Two options do most of the work: keyed_groups builds groups automatically from tags or attributes - so an instance tagged role: web lands in a role_web group with no manual step - and compose builds host variables from discovered fields.

An inventory script is the older, universal escape hatch: any executable honouring the --list/--host JSON contract from Phase 1. It is the right tool when your source of truth is an internal CMDB with no plugin.

Multiple sources can be combined. Pointing inventory at a directory makes Ansible merge every source inside it, so a static file of fixed infrastructure can sit alongside a dynamic cloud source.

The operational risk is availability. If the API is down or credentials expired, you get an empty inventory - and an empty inventory produces a completely green run that did nothing at all. Enable inventory caching, and assert a minimum host count before anything important runs.

Analogy: A static inventory is a printed org chart from March. Dynamic inventory is looking people up in the live directory - which is right until the directory service is down and you conclude the company has no employees.

A worked example

# inventory/cloud.yml - an inventory PLUGIN config
plugin: amazon.aws.aws_ec2
regions:
  - eu-west-1
filters:
  tag:Environment: production
keyed_groups:
  - key: tags.Role          # tag Role=web  ->  group role_web
    prefix: role
  - key: placement.availability_zone
    prefix: az
compose:
  ansible_host: private_ip_address
cache: true
cache_plugin: jsonfile
cache_connection: /var/tmp/ansible_inventory
cache_timeout: 3600

# Merge several sources by pointing at a DIRECTORY
# ansible.cfg:
#   inventory = ./inventory
# inventory/
# ├── static.ini     # fixed on-prem hardware
# └── cloud.yml      # live cloud instances

# Guard against an empty inventory in CI
- ansible.builtin.assert:
    that: groups['production'] | length >= 4
    fail_msg: "Inventory returned too few hosts - refusing to run."
  run_once: true

ansible-inventory --graph and --list remain the tools that tell you what Ansible actually built. Run them after any inventory change; the difference between what you configured and what you got is where most dynamic-inventory bugs live.

When migrating from static to dynamic, run both side by side for a while: generate the dynamic inventory, diff its host list against the static one, and investigate every difference. Some will be stale static entries; some will be gaps in your filters. Both are worth knowing before you delete the file.

Warning: An inventory source that fails and returns zero hosts produces a fully successful run that changed nothing. Assert a minimum host count, or automation will report success for doing nothing.

Goal: Put this to work in the ansible-dynamic-inventory lab. Open /labs/ansible, pick ansible-dynamic-inventory, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.