Backpropagation is the standard method by which neural networks learn. After each prediction, the error is measured and propagated backwards through the network's layers, showing how much each weight contributed to it. An optimisation method such as gradient descent then adjusts the weights step by step. Without backpropagation, training today's language and image models would be practically impossible.
Gradient descent is the optimisation method used to train most AI models. You measure how large the model’s error is, calculate in which direction the parameters must change to reduce it, and then take a small step in that direction – again and again. It is like a hiker in fog who keeps walking downhill to reach the valley. The step size (learning rate) determines speed and stability.
Artificial neural networks (ANNs) are computational models made up of many connected units (“neurons”) arranged in layers that process inputs via weighted connections. During training, the weights are adjusted step by step using examples, usually by backpropagation. ANNs are the foundation of deep learning and power image recognition, speech recognition and large language models. Despite their biological inspiration, they work far more simply than the human brain.
Activation functions determine how strongly a neuron in an artificial neural network passes on its signal. They introduce non-linearity – only then can the network represent complex relationships such as language or images. Common examples are ReLU, which sets negative values to zero, sigmoid for probabilities between 0 and 1, and GELU, used in many transformer models. The choice affects training speed and stability.
Cross-entropy is a measure from information theory that indicates how far a predicted probability distribution deviates from the actual one. In machine learning, it is the most common loss function for classification tasks and also for language models that predict the next token. It penalises confident wrong predictions especially heavily: assigning 99% to the wrong class yields a much higher loss than assigning 60%.
