A recurrent neural network processes data in order and passes an internal state from each step to the next – a kind of short-term memory. This makes it suitable for sequences such as text, speech or time series, for example to predict the next word or the next sensor reading. Simple RNNs easily forget information from far back; variants such as LSTM and GRU mitigate this. For language models, RNNs have largely been superseded by transformers.
Long Short-Term Memory (LSTM) is a type of recurrent neural network that can retain information across long sequences. So-called gates decide at each step what to store, forget or output, which mitigates the vanishing gradient problem. For many years LSTMs shaped speech recognition, translation and time-series forecasting. In language processing they have largely been replaced by transformers, but they remain common for sensor data.
A bidirectional LSTM (BiLSTM) is a recurrent neural network that reads a sequence both forwards and backwards and combines the two views. This means the context that follows each word is taken into account as well. Before transformers took over, it was a standard choice for tasks such as part-of-speech tagging, named entity recognition and speech recognition. For very long texts and large datasets, transformers now usually perform better.
Sequence modelling covers methods that process and predict data in a fixed order, where the position of each element matters. Typical sequences are text, spoken language, time series such as sales or sensor readings, and DNA segments. Applications range from translation and text generation to speech recognition and sales forecasting. Hidden Markov models and recurrent networks have been used for this; today transformer models dominate for language.
The transformer is a neural network architecture introduced in 2017 in the research paper “Attention Is All You Need”. At its core is the attention mechanism: each word is weighted in relation to all other words in a text, and the computation can be parallelised efficiently. Transformers underpin today's language models such as GPT, Claude and BERT and are also used for images and audio.
