Dropout is a regularisation technique used when training neural networks: in each training step a random share of neurons is temporarily switched off. This prevents the network from relying on individual connections, so it learns more robust patterns. It reduces overfitting, i.e. merely memorising the training data. When the finished model is used later, all neurons are active again.
Overfitting occurs when a model memorises the training data, including random quirks and noise, instead of capturing generally valid patterns. It then appears very accurate on familiar data but performs poorly on new data. A large gap between training and test error is a typical sign. Countermeasures include more or more varied data, simpler models, regularisation, dropout and cross-validation.
Regularisation covers techniques that stop a model from fitting its training data too closely during training (overfitting). Examples include penalising large weights (L1 or L2 regularisation), randomly switching off neurons (dropout) or stopping training at the right moment (early stopping). The aim is a simpler model that performs more reliably on new, unseen data, even if it scores slightly lower on the training data.
Artificial neural networks (ANNs) are computational models made up of many connected units (“neurons”) arranged in layers that process inputs via weighted connections. During training, the weights are adjusted step by step using examples, usually by backpropagation. ANNs are the foundation of deep learning and power image recognition, speech recognition and large language models. Despite their biological inspiration, they work far more simply than the human brain.
Batch normalisation is a technique for training deep neural networks that rescales a layer's intermediate values for each batch of data to a consistent mean and spread. This makes training more stable and often faster, and the network becomes less sensitive to its initial values. Introduced in 2015, it is common in image recognition networks; transformers usually rely on the related layer normalisation.
