Batch normalisation is a technique for training deep neural networks that rescales a layer's intermediate values for each batch of data to a consistent mean and spread. This makes training more stable and often faster, and the network becomes less sensitive to its initial values. Introduced in 2015, it is common in image recognition networks; transformers usually rely on the related layer normalisation.
Deep learning is a branch of machine learning that uses artificial neural networks with many layers. Such networks learn from large amounts of data which features matter – from edges in images to relationships of meaning in texts. Deep learning underpins today's language models, image and speech recognition and machine translation, and usually requires considerable computing power and training data.
Dropout is a regularisation technique used when training neural networks: in each training step a random share of neurons is temporarily switched off. This prevents the network from relying on individual connections, so it learns more robust patterns. It reduces overfitting, i.e. merely memorising the training data. When the finished model is used later, all neurons are active again.
Regularisation covers techniques that stop a model from fitting its training data too closely during training (overfitting). Examples include penalising large weights (L1 or L2 regularisation), randomly switching off neurons (dropout) or stopping training at the right moment (early stopping). The aim is a simpler model that performs more reliably on new, unseen data, even if it scores slightly lower on the training data.
Gradient descent is the optimisation method used to train most AI models. You measure how large the model’s error is, calculate in which direction the parameters must change to reduce it, and then take a small step in that direction – again and again. It is like a hiker in fog who keeps walking downhill to reach the valley. The step size (learning rate) determines speed and stability.
