Bagging (short for bootstrap aggregating) is an ensemble method in machine learning: many models are trained on randomly drawn samples of the training data, and their predictions are averaged or combined by majority vote. This makes results more stable and reduces overfitting. The best-known example is the random forest, which combines many decision trees – for instance to predict customer churn or prices.
Random forest regression predicts numerical values by combining many decision trees. Each tree is trained on a random sample of the data and a random selection of features; the result is the average of all the trees' predictions. This makes the method more robust and less prone to overfitting than a single tree. It is used, for example, for sales forecasts, price estimates or energy demand – often as a solid starting point for tabular data.
Boosting is an ensemble method in machine learning in which many simple models are trained one after another, each new model focusing on the cases its predecessors got wrong. Together they form a much more accurate overall model. Well-known variants are AdaBoost and gradient boosting, with libraries such as XGBoost or LightGBM. Boosting is particularly strong on tabular data, for example credit risk, sales or churn forecasts.
A decision tree is a model that represents a decision as a sequence of simple yes/no questions: each branch tests one feature and each leaf gives an outcome. In machine learning, such trees are learnt automatically from data and used for classification or forecasting. Their main strength is transparency – for instance, showing why a loan application was flagged based on income, term and payment history.
Overfitting occurs when a model memorises the training data, including random quirks and noise, instead of capturing generally valid patterns. It then appears very accurate on familiar data but performs poorly on new data. A large gap between training and test error is a typical sign. Countermeasures include more or more varied data, simpler models, regularisation, dropout and cross-validation.
