Gradient descent is the optimisation method used to train most AI models. You measure how large the model’s error is, calculate in which direction the parameters must change to reduce it, and then take a small step in that direction – again and again. It is like a hiker in fog who keeps walking downhill to reach the valley. The step size (learning rate) determines speed and stability.
Backpropagation is the standard method by which neural networks learn. After each prediction, the error is measured and propagated backwards through the network's layers, showing how much each weight contributed to it. An optimisation method such as gradient descent then adjusts the weights step by step. Without backpropagation, training today's language and image models would be practically impossible.
In machine learning, convergence means that a training procedure approaches a stable state: the loss barely decreases any more and the model parameters change only minimally. Whether and how quickly a model converges depends on factors such as learning rate, data quality and model architecture. If the learning rate is too high, the loss jumps around; if it is too low, training takes unnecessarily long. Convergence does not guarantee a good model, as it may still be overfitted.
Cross-entropy is a measure from information theory that indicates how far a predicted probability distribution deviates from the actual one. In machine learning, it is the most common loss function for classification tasks and also for language models that predict the next token. It penalises confident wrong predictions especially heavily: assigning 99% to the wrong class yields a much higher loss than assigning 60%.
Hyperparameters are settings fixed before a model is trained, such as the learning rate, the number of layers or the strength of regularisation. Hyperparameter optimisation systematically searches for the combination that performs best on validation data, for instance through grid search, random search or Bayesian optimisation. Well-chosen values improve accuracy and reduce over- or underfitting, but the search costs computing time.
