Count Tokens and Estimate Cost

Text usage has units

Back in the very first module of this track, a token was introduced as the small chunk of text an LLM actually reads and generates - a whole common word, a piece of punctuation, or a fragment of a longer word. It's worth returning to that concept here because tokens are also the unit language models get measured and billed in. A single English word might be one token or several, and punctuation quietly consumes tokens too, which is why simply counting words is not the same as counting tokens. Input tokens cover everything sent to the model - instructions, prior conversation history, and any retrieved context stitched in by a RAG pipeline. Output tokens are what the model actually generates in response.

Hosted providers commonly price input and output tokens separately, often quoted per million tokens, and often at different rates from each other (output tokens are frequently more expensive than input tokens, since generating text costs more compute than reading it). A simple cost estimate looks like this:

estimated_cost = (input_tokens * input_price + output_tokens * output_price) / 1_000_000

It's worth recording which version of a provider's price table you used for that calculation, since prices do change over time and an old estimate recalculated with new prices would be misleading. Local inference, like everything used in this platform, has no per-call bill from a provider at all - but building the same cost-estimation habit here still teaches real budgeting discipline, and lets you meaningfully compare "how expensive would this pattern of usage be if I were paying a hosted provider for it."

A word of caution on how to actually count tokens: production systems should use the real tokenizer, or the token counts the model provider itself reports back, rather than approximating. Splitting text on whitespace and counting the pieces is a transparent, easy-to-explain approximation at best - useful for a rough estimate - but counting plain characters is not a valid substitute at all, since words vary enormously in length and a character count bears little relation to an actual token count.

Scenario: A team notices its daily inference bill has doubled almost overnight and initially assumes user traffic simply grew. Pulling up the traces shows output token counts stayed essentially flat - but the retrieved context being stuffed into each prompt had roughly tripled in size, revealing that growth in the RAG pipeline's retrieval step, not user traffic at all, was the real cause of the cost increase.

Goal: Instrument the lab's service so every single request consistently records matching, internally-consistent input-token, output-token, total-token, and estimated-cost fields - the exact fields shown in the trace example from the previous section.