Boosting is an ensemble method in machine learning in which many simple models are trained one after another, each new model focusing on the cases its predecessors got wrong. Together they form a much more accurate overall model. Well-known variants are AdaBoost and gradient boosting, with libraries such as XGBoost or LightGBM. Boosting is particularly strong on tabular data, for example credit risk, sales or churn forecasts.
Gradient boosting is a machine learning technique that combines many simple models – usually small decision trees – one after another. Each new tree is trained specifically to reduce the remaining errors of the previous ones. The result is highly accurate models for tabular data, for example for credit risk, sales forecasts or churn probability. Well-known implementations are XGBoost, LightGBM and CatBoost; careful tuning is needed to avoid overfitting.
XGBoost (Extreme Gradient Boosting) is an open-source library that implements gradient boosting with decision trees particularly efficiently. Many small trees are trained one after another, each correcting the errors of the previous ones, while regularisation helps prevent overfitting. XGBoost is regarded as a proven choice for tabular data, for example in credit risk scoring, churn prediction or sales forecasting.
LightGBM is an open-source library for gradient boosting with decision trees, originally developed by Microsoft. It is designed for speed and low memory use, among other things through histogram-based training and trees that grow leaf-wise at the most promising leaf. LightGBM is particularly suited to large tabular datasets, for example sales forecasts, churn prediction or fraud detection, and is often compared with XGBoost.
Bagging (short for bootstrap aggregating) is an ensemble method in machine learning: many models are trained on randomly drawn samples of the training data, and their predictions are averaged or combined by majority vote. This makes results more stable and reduces overfitting. The best-known example is the random forest, which combines many decision trees – for instance to predict customer churn or prices.
