Training error indicates how far a model's predictions deviate from the correct values on exactly the data it was trained on. On its own it says little: what matters is the comparison with the generalisation error on new data. A high training error points to underfitting; a very low training error combined with a high error on test data usually signals overfitting.
The generalisation error measures how far off an AI model is on new, previously unseen data – as opposed to its error on the training data. A large gap between the two points to overfitting: the model has memorised examples instead of learning general patterns. It is estimated using held-out test data or cross-validation. It is the figure that really counts, because in practice a model almost only meets new cases.
Overfitting occurs when a model memorises the training data, including random quirks and noise, instead of capturing generally valid patterns. It then appears very accurate on familiar data but performs poorly on new data. A large gap between training and test error is a typical sign. Countermeasures include more or more varied data, simpler models, regularisation, dropout and cross-validation.
Underfitting occurs when a model is too simple to capture the relationships in the data. It then makes many errors on the training data as well as on new data, for example when a straight line is used to represent a clearly curved pattern. Remedies include a more powerful model, more informative features, longer training or less regularisation; the opposite problem is overfitting.
Cross-validation is a method for checking how well a model generalises to new, unseen data. In k-fold cross-validation, the data is split into k parts; the model is trained k times on k−1 parts and tested on the remaining part each time, after which the results are averaged. This helps detect overfitting and compare models or settings fairly, especially when little data is available.
