Tokens Are Cost and Capacity

Context is not free memory

Input tokens consume prefill work and context space; output tokens consume sequential decode time and are often priced differently by hosted providers. Retrieved documents, conversation history, examples, and system policy all compete for one context window. Larger context can add useful evidence or dilute attention with irrelevant text.

A cost model multiplies token counts by versioned prices, then adds non-model infrastructure and retry/fallback amplification. Local inference has no per-token invoice but still consumes CPU time, memory, energy, and finite concurrency.

daily cost = requests × (avg input tokens × input rate
                       + avg output tokens × output rate)
             + embedding, retrieval, storage, and serving costs

Scenario: RAG top-k rises from 4 to 20. Accuracy barely moves, while prompt tokens triple, TTFT grows, and more answers exceed context. Re-ranking or better chunking is preferable to indiscriminately adding context.

Warning: Never quote current hosted prices from memory in operational code. Store provider, model, currency, region if relevant, and effective date with every rate table.