LSTM Gates Manage Memory

Learn what to store, forget, and reveal

A long short-term memory (LSTM) network adds a dedicated cell state and learned gates. The forget gate decides what old memory to remove, the input gate decides what new information to write, and the output gate decides what part of memory to expose as hidden state.

old cell -- forget gate --+
new candidate -- input gate+--> new cell --> output gate --> hidden state

The additive cell-state path lets gradients travel more directly than through repeated plain-RNN transformations, improving long-term learning. A gated recurrent unit (GRU) provides a simpler related design with fewer gates and parameters.

Analogy: An LSTM is a notebook with three editors: one erases stale notes, one writes relevant facts, and one chooses what to read aloud now.

Tip: RNNs and LSTMs remain useful for streaming and modest sequential tasks, but attention-based Transformers dominate many language workloads because they connect distant tokens more directly and parallelize training better.