Skip to content
1 Algorithm History
In 1986, Elman and others proposed the Recurrent Neural Network for processing sequential data. Just as Convolutional Neural Networks are specifically designed for processing two-dimensional data (such as images), Recurrent Neural Networks are specialized for handling sequential information. Recurrent networks can be extended to longer sequences, and most recurrent neural networks can handle variable-length sequences. The emergence of recurrent neural networks addressed the limitations of traditional neural networks in processing sequential information.
In 1997, Hochreiter and Schmidhuber proposed the Long Short-Term Memory (LSTM) unit to solve the vanishing gradient problem in standard recurrent neural networks. The standard recurrent neural network structure has a limited range of stored contextual information, which restricts the application of RNNs. LSTM-type RNNs replace the neuron nodes in the standard structure with LSTM units, which use input gates, output gates, and forget gates to control the transmission of sequential information, thus achieving a greater range of context information retention and transmission.
In 1998, Williams and Zipser introduced a training algorithm for recurrent neural networks called Backpropagation Through Time (BPTT). The essence of the BPTT algorithm is to unfold the recurrent neural network according to the time sequence, resulting in a network containing N (time steps) hidden units and one output unit, and then update the connection weights of the neural network using backpropagation of error.
In 2001, Gers and Schmidhuber proposed a significant optimization model for LSTM-type RNNs, incorporating peephole connections into the traditional LSTM units. The LSTM-type RNN model with peephole connections is one of the most popular models of recurrent neural networks, as peephole connections further enhance the LSTM unit’s ability to process sequential information with long time interval correlations. In 2005, Graves successfully applied LSTM-type RNNs to speech processing; in 2007, Hochreiter applied LSTM-type RNNs to bioinformatics research.
2 Algorithm Overview
Recurrent Neural Networks (RNNs) are a commonly used neural network structure that originated from the Hopfield network proposed by Saratha Sathasivam in 1982. Its unique cyclic concept and its most important structure—the Long Short-Term Memory network—enable it to perform well in processing and predicting sequential data.
The idea behind RNNs is to utilize sequential information. In traditional neural networks, we assume that all inputs (and outputs) are independent of each other. However, this is a very poor assumption for many tasks. If you want to predict the next word in a sentence, it is better to know what words came before it. RNNs are called
