Regularisation covers techniques that stop a model from fitting its training data too closely during training (overfitting). Examples include penalising large weights (L1 or L2 regularisation), randomly switching off neurons (dropout) or stopping training at the right moment (early stopping). The aim is a simpler model that performs more reliably on new, unseen data, even if it scores slightly lower on the training data.
Overfitting occurs when a model memorises the training data, including random quirks and noise, instead of capturing generally valid patterns. It then appears very accurate on familiar data but performs poorly on new data. A large gap between training and test error is a typical sign. Countermeasures include more or more varied data, simpler models, regularisation, dropout and cross-validation.
Dropout is a regularisation technique used when training neural networks: in each training step a random share of neurons is temporarily switched off. This prevents the network from relying on individual connections, so it learns more robust patterns. It reduces overfitting, i.e. merely memorising the training data. When the finished model is used later, all neurons are active again.
The generalisation error measures how far off an AI model is on new, previously unseen data – as opposed to its error on the training data. A large gap between the two points to overfitting: the model has memorised examples instead of learning general patterns. It is estimated using held-out test data or cross-validation. It is the figure that really counts, because in practice a model almost only meets new cases.
Underfitting occurs when a model is too simple to capture the relationships in the data. It then makes many errors on the training data as well as on new data, for example when a straight line is used to represent a clearly curved pattern. Remedies include a more powerful model, more informative features, longer training or less regularisation; the opposite problem is overfitting.
