Sequence modelling covers methods that process and predict data in a fixed order, where the position of each element matters. Typical sequences are text, spoken language, time series such as sales or sensor readings, and DNA segments. Applications range from translation and text generation to speech recognition and sales forecasting. Hidden Markov models and recurrent networks have been used for this; today transformer models dominate for language.
A recurrent neural network processes data in order and passes an internal state from each step to the next – a kind of short-term memory. This makes it suitable for sequences such as text, speech or time series, for example to predict the next word or the next sensor reading. Simple RNNs easily forget information from far back; variants such as LSTM and GRU mitigate this. For language models, RNNs have largely been superseded by transformers.
Long Short-Term Memory (LSTM) is a type of recurrent neural network that can retain information across long sequences. So-called gates decide at each step what to store, forget or output, which mitigates the vanishing gradient problem. For many years LSTMs shaped speech recognition, translation and time-series forecasting. In language processing they have largely been replaced by transformers, but they remain common for sensor data.
The transformer is a neural network architecture introduced in 2017 in the research paper “Attention Is All You Need”. At its core is the attention mechanism: each word is weighted in relation to all other words in a text, and the computation can be parallelised efficiently. Transformers underpin today's language models such as GPT, Claude and BERT and are also used for images and audio.
A hidden Markov model (HMM) is a statistical model for sequences in which the actual states are hidden and can only be inferred from observable signals. It describes the probability of one state following another and of each state producing a given observation. HMMs were long the standard in speech recognition and part-of-speech tagging; today they are still used, for example, in bioinformatics and time-series analysis.
