Depth, Width, and Capacity

More units create more possible functions

Width is the number of units in a layer; depth is the number of successive learned layers. Increasing either usually increases parameter count and representational capacity. Wider networks can learn many features in parallel; deeper networks can compose features across stages.

Capacity brings costs: memory, inference time, training data needs, optimization difficulty, and overfitting risk. A smaller network may generalize better or meet latency requirements even when a larger one achieves lower training loss.

input 4 -> hidden 8 -> output 1
weights: 4×8 + 8×1 = 40, plus biases
input 4000 -> hidden 8000 -> ... becomes a very different budget

Tip: Count parameters and activation memory, not just layers. Sequence length and batch size can dominate runtime memory even when weights stay fixed.

Note: This platform intentionally teaches architecture without running deep-learning frameworks. The offline CPU image is sized for classical ML and pre-trained LLM inference, not neural-network training.