CPU and Memory Capacity

Budget the whole process

It's tempting to think "the model file is 400 MB, so I need 400 MB of RAM" - but the model file is only one piece of what actually sits in memory while a model is running. On top of the loaded weights, the runtime also needs temporary working areas called buffers, the memory used by the Python process itself, any vector indexes your application built (covered in the RAG modules), and ordinary operating-system memory pages. It also maintains a key-value cache (KV cache): a record of calculations from earlier tokens, kept around so the model doesn't have to redo that work from scratch for every single new token it generates. A longer context window, or more requests running at the same time, directly makes that cache bigger - which is one reason context length and concurrency both show up repeatedly as "the knob to turn" throughout this track.

On this shared training platform, that whole budget is deliberately capped: Python-only AI labs (the ones that never load an actual language model) get a 1 gibibyte (GiB) memory ceiling, while the warmup sandbox and the generation-focused labs - the ones that can load Qwen2.5:0.5b or SmolLM2-135M - get 2 GiB, since they need room for both the model and its KV cache. Every tier is additionally limited to half of one CPU core and one model request running at a time. These aren't arbitrary numbers; they exist specifically so that one learner's experiment can't starve the shared PostgreSQL database, Redis cache, or every other learner's container of the memory they need.

Warning: If a lab or application hits an out-of-memory error, the fix is never to quietly swap in a larger model and hope the extra headroom papers over the problem. Instead, reduce the actual cause - a smaller model, a shorter context window, lower concurrency, or a lighter dependency footprint - and re-measure to confirm the fix actually worked.