Why Plain RNNs Forget

Distant evidence crosses too many steps

Training an RNN uses backpropagation through time, applying the backward chain across every recurrent step. Signals from early tokens are multiplied repeatedly before reaching them, creating vanishing or exploding gradients. As a result, plain RNNs struggle to learn long-range dependencies.

The database credentials, stored in the sealed envelope ... [200 tokens] ... were expired.
^ early subject must influence interpretation far later

The hidden state is also a fixed-size bottleneck asked to compress an arbitrarily long history. Recent inputs can overwrite earlier information, even when that early detail becomes important later.

Scenario: A sequence model predicts a closing delimiter correctly when it is five tokens away but fails when 200 tokens separate the pair. Short training examples hid the memory limitation.

Warning: Randomly splitting overlapping time windows can leak nearly identical sequences across train and test. Split by time or source group before window generation when necessary.