NLTK (Natural Language Toolkit) is an open-source Python library for natural language processing, created at the University of Pennsylvania in the early 2000s. It provides tools for tokenisation, stemming, part-of-speech tagging and simple classification, as well as access to numerous text corpora. NLTK is used mainly in teaching and research; production applications often rely on libraries such as spaCy or on transformer models.
spaCy is an open-source Python library for natural language processing, developed by the Berlin-based company Explosion. It splits texts into words and sentences, determines parts of speech and base forms, analyses sentence structure and recognises named entities such as people, companies or places. Pre-trained models are available for many languages, including German. spaCy is designed for speed and for use in real applications, for example to pre-process large volumes of documents.
Natural language processing is the branch of AI that enables computers to analyse, understand and generate human language in text or speech. It covers tasks such as translation, classification, summarisation, sentiment analysis and question answering. Rules and statistical methods once dominated; today the field is shaped by large language models.
Part-of-speech tagging automatically assigns each word in a sentence its grammatical category, such as noun, verb, adjective or preposition. Because many words are ambiguous (“book” as a noun or a verb, “light” as an adjective or a noun), the methods take the sentence context into account. POS tagging is a classic pre-processing step for parsing, information extraction or spell checking and is offered by libraries such as spaCy and NLTK.
Gensim is an open-source Python library for analysing large collections of text. It is known for topic models such as LDA, for training word vectors (for instance Word2Vec, FastText and Doc2Vec) and for comparing documents by similarity. Because Gensim processes data in chunks, it can also handle corpora that do not fit into memory. A typical use: automatically grouping thousands of customer comments by recurring themes.
