Overfitting occurs when a model memorises the training data, including random quirks and noise, instead of capturing generally valid patterns. It then appears very accurate on familiar data but performs poorly on new data. A large gap between training and test error is a typical sign. Countermeasures include more or more varied data, simpler models, regularisation, dropout and cross-validation.
Underfitting occurs when a model is too simple to capture the relationships in the data. It then makes many errors on the training data as well as on new data, for example when a straight line is used to represent a clearly curved pattern. Remedies include a more powerful model, more informative features, longer training or less regularisation; the opposite problem is overfitting.
Regularisation covers techniques that stop a model from fitting its training data too closely during training (overfitting). Examples include penalising large weights (L1 or L2 regularisation), randomly switching off neurons (dropout) or stopping training at the right moment (early stopping). The aim is a simpler model that performs more reliably on new, unseen data, even if it scores slightly lower on the training data.
Cross-validation is a method for checking how well a model generalises to new, unseen data. In k-fold cross-validation, the data is split into k parts; the model is trained k times on k−1 parts and tested on the remaining part each time, after which the results are averaged. This helps detect overfitting and compare models or settings fairly, especially when little data is available.
The generalisation error measures how far off an AI model is on new, previously unseen data – as opposed to its error on the training data. A large gap between the two points to overfitting: the model has memorised examples instead of learning general patterns. It is estimated using held-out test data or cross-validation. It is the figure that really counts, because in practice a model almost only meets new cases.
