BERT (Bidirectional Encoder Representations from Transformers) is a language model introduced by Google in 2018. Unlike earlier models, it considers the context to the left and right of a word at the same time, which helps it resolve ambiguity. BERT is designed for understanding text rather than writing longer passages. It is used for search, text classification, named entity recognition and question answering and has shaped many successor models.
The transformer is a neural network architecture introduced in 2017 in the research paper “Attention Is All You Need”. At its core is the attention mechanism: each word is weighted in relation to all other words in a text, and the computation can be parallelised efficiently. Transformers underpin today's language models such as GPT, Claude and BERT and are also used for images and audio.
Natural language understanding is the part of language processing that captures the meaning of an utterance: what intent lies behind it, which people, places or dates are mentioned, and what is its tone? Together with natural language generation (NLG), NLU forms one of the two sides of NLP. One example is a customer service system that recognises “My parcel still hasn't arrived” as a delivery enquiry.
Word embeddings represent words as numerical vectors so that words with similar meanings lie close together in vector space, such as “car” and “vehicle”. Methods such as Word2Vec or GloVe learn these vectors from large amounts of text. They were an important step in language processing, for example for search or classification; today's language models use contextual embeddings in which a word is represented differently depending on the sentence.
GPT (generative pre-trained transformer) refers to a series of large language models developed by OpenAI and based on the transformer architecture. The models are first pre-trained on vast amounts of text to predict the next token and then further trained for dialogue and following instructions. The first GPT model appeared in 2018; later generations underpin ChatGPT. In everyday speech, ‘GPT’ is often used for language models in general.
