Making Large Runs Fast

Scenario: A nightly compliance run against 600 hosts takes four hours. Most of that time is spent gathering facts that no task in the playbook ever reads.

New words, in plain English

• forks - how many hosts Ansible works on in parallel. Default 5.
• Pipelining - sending module code over an existing SSH session instead of copying files - fewer round trips.
• ControlPersist - keeping an SSH connection open and reusing it for later tasks.
• Fact caching - storing gathered facts so subsequent runs reuse them.
• strategy: free - let each host proceed at its own pace instead of waiting at each task.

Ansible's default settings are conservative. On a large estate a handful of changes usually deliver most of the available speedup.

forks defaults to 5, meaning only five hosts are worked on at once. Raising it to 25-50 is often the single biggest win. The limit is the control node's CPU and file descriptors, not the managed hosts.

Pipelining (pipelining = True) sends module code down an existing SSH session instead of copying a file and then executing it, removing several round trips per task. It requires requiretty to be disabled in sudoers on the targets, which is the default on modern systems.

SSH connection reuse via ControlMaster and ControlPersist keeps the connection open between tasks. Ansible sets sensible defaults; raising ControlPersist to 60s or more helps long plays.

Fact gathering is usually the biggest single cost, and the fix is the one from Phase 2: gather_facts: false where facts are unused, gather_subset where only cheap ones are needed, and fact caching so repeat runs reuse them.

strategy: free lets each host run ahead independently rather than waiting for the slowest host at every task. It is a large win on heterogeneous fleets, and it is unsafe for any play where ordering across hosts matters.

Analogy: The default settings are a checkout with five tills open and every customer's shopping weighed at the door. Open more tills, stop weighing shopping nobody asked about, and let people who are ready leave without waiting for the slowest trolley.

A worked example

# ansible.cfg - the settings that matter on a large estate
[defaults]
forks = 30                       # default is 5
gathering = smart                # gather only if not already cached
fact_caching = jsonfile
fact_caching_connection = /var/tmp/ansible_facts
fact_caching_timeout = 7200

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=120s

# In the playbook
- name: Compliance sweep
  hosts: all
  gather_facts: true
  gather_subset: ['!all', 'min']    # skip hardware and network enumeration
  strategy: free                     # hosts proceed independently
  tasks:
    - ...

# Measure before and after - guessing is not optimisation
# ansible.cfg:
#   callbacks_enabled = ansible.posix.profile_tasks

Measure rather than guess. The profile_tasks callback prints per-task timings and a summary of the slowest tasks, which usually makes the answer obvious - and it is frequently not what people expected. A single command task doing a package-manager cache update can dominate a run that everyone assumed was slow because of SSH.

gathering = smart is a good default: it gathers facts only for hosts that do not already have valid cached facts, so the first run of the day pays the cost and the rest do not.

Be careful with strategy: free. It removes the implicit synchronisation between hosts at each task, so anything that assumed all hosts had finished task N before any started task N+1 will break. It pairs badly with serial and with cross-host hostvars reads that depend on ordering.

Tip: Enable profile_tasks before changing anything. Optimising the wrong task is the most common way to spend a day and save nothing.

Goal: Put this to work in the ansible-facts-performance lab. Open /labs/ansible, pick ansible-facts-performance, and fix the real broken project - Ansible really does SSH into four managed hosts and converge them.