RNN’s are Schmidhuber’s Revenge

Jürgen Schmidhuber, the father of Long short-term memory (LSTM), was probably right when he said recurrent neural networks (RNNs) are all we need. While Transformers, using attention, dominate generative AI right now, they still struggle when dealing with long sequences.

But researchers from Borealis AI,  the Ontario-based research firm, decided to revisit RNNs to see if they can solve some current problems with LLMs. Led by Yoshua Bengio, one of the godfathers of deep learning, Borealis AI believes that RNNs introduced in 2015 were slower earlier because they needed to go through the backpropagation (BPTT) method, something that Schmidhuber has frequently claimed credit for introducing. 

Were RNNs All We Needed?

The researchers asked this question to revive traditional RNNs, including LSTMs and Gated Recurrent Units (GRUs). They concluded that by removing the hidden state dependencies from their input, forgetting them, and updating gates, LSTMs and GRUs no longer need BPTT and can be efficiently trained in parallel.