PaLM is a large language model from Google, unveiled in 2022 with 540 billion parameters and trained across many TPU chips at once using Google's Pathways training system. PaLM 2 followed in 2023 and powered, among other things, the Bard chatbot. Google has since replaced PaLM with its Gemini models, and the PaLM developer API was shut down in August 2024. Today PaLM matters mainly as a milestone in the development of large language models.
A large language model (LLM) is a neural network trained on very large amounts of text, enabling it to understand and generate language. It works by calculating the most likely next token, step by step. LLMs write, summarise, translate, answer questions and generate code; well-known examples include Claude, GPT, Gemini and Llama. Without access to current sources, they only know what was in their training data.
The transformer is a neural network architecture introduced in 2017 in the research paper “Attention Is All You Need”. At its core is the attention mechanism: each word is weighted in relation to all other words in a text, and the computation can be parallelised efficiently. Transformers underpin today's language models such as GPT, Claude and BERT and are also used for images and audio.
A foundation model is a large AI model pre-trained on very extensive, wide-ranging data that can then be used for many different tasks. The term was coined at Stanford University in 2021. Well-known examples include large language models such as Claude or GPT, as well as image models. Businesses usually build their applications on such models rather than training their own from scratch; the EU AI Act refers to general-purpose AI models.
GPT (generative pre-trained transformer) refers to a series of large language models developed by OpenAI and based on the transformer architecture. The models are first pre-trained on vast amounts of text to predict the next token and then further trained for dialogue and following instructions. The first GPT model appeared in 2018; later generations underpin ChatGPT. In everyday speech, ‘GPT’ is often used for language models in general.
