Gensim is an open-source Python library for analysing large collections of text. It is known for topic models such as LDA, for training word vectors (for instance Word2Vec, FastText and Doc2Vec) and for comparing documents by similarity. Because Gensim processes data in chunks, it can also handle corpora that do not fit into memory. A typical use: automatically grouping thousands of customer comments by recurring themes.
Word embeddings represent words as numerical vectors so that words with similar meanings lie close together in vector space, such as “car” and “vehicle”. Methods such as Word2Vec or GloVe learn these vectors from large amounts of text. They were an important step in language processing, for example for search or classification; today's language models use contextual embeddings in which a word is represented differently depending on the sentence.
NLTK (Natural Language Toolkit) is an open-source Python library for natural language processing, created at the University of Pennsylvania in the early 2000s. It provides tools for tokenisation, stemming, part-of-speech tagging and simple classification, as well as access to numerous text corpora. NLTK is used mainly in teaching and research; production applications often rely on libraries such as spaCy or on transformer models.
spaCy is an open-source Python library for natural language processing, developed by the Berlin-based company Explosion. It splits texts into words and sentences, determines parts of speech and base forms, analyses sentence structure and recognises named entities such as people, companies or places. Pre-trained models are available for many languages, including German. spaCy is designed for speed and for use in real applications, for example to pre-process large volumes of documents.
Natural language processing is the branch of AI that enables computers to analyse, understand and generate human language in text or speech. It covers tasks such as translation, classification, summarisation, sentiment analysis and question answering. Rules and statistical methods once dominated; today the field is shaped by large language models.
