Gradient boosting is a machine learning technique that combines many simple models – usually small decision trees – one after another. Each new tree is trained specifically to reduce the remaining errors of the previous ones. The result is highly accurate models for tabular data, for example for credit risk, sales forecasts or churn probability. Well-known implementations are XGBoost, LightGBM and CatBoost; careful tuning is needed to avoid overfitting.
XGBoost (Extreme Gradient Boosting) is an open-source library that implements gradient boosting with decision trees particularly efficiently. Many small trees are trained one after another, each correcting the errors of the previous ones, while regularisation helps prevent overfitting. XGBoost is regarded as a proven choice for tabular data, for example in credit risk scoring, churn prediction or sales forecasting.
LightGBM is an open-source library for gradient boosting with decision trees, originally developed by Microsoft. It is designed for speed and low memory use, among other things through histogram-based training and trees that grow leaf-wise at the most promising leaf. LightGBM is particularly suited to large tabular datasets, for example sales forecasts, churn prediction or fraud detection, and is often compared with XGBoost.
Boosting is an ensemble method in machine learning in which many simple models are trained one after another, each new model focusing on the cases its predecessors got wrong. Together they form a much more accurate overall model. Well-known variants are AdaBoost and gradient boosting, with libraries such as XGBoost or LightGBM. Boosting is particularly strong on tabular data, for example credit risk, sales or churn forecasts.
A decision tree is a model that represents a decision as a sequence of simple yes/no questions: each branch tests one feature and each leaf gives an outcome. In machine learning, such trees are learnt automatically from data and used for classification or forecasting. Their main strength is transparency – for instance, showing why a loan application was flagged based on income, term and payment history.
