EN
Sign inGet started
Glossary

AI Quality Assurance Specialist

What is AI Quality Assurance Specialist?

An AI quality assurance specialist checks whether AI applications work reliably, correctly and appropriately. The role includes test questions and benchmarks, comparing different models, analysing user ratings, and spotting hallucinations and deterioration over time. For example, before switching to a new model, 50 typical customer enquiries are answered by both models and compared against fixed criteria.

How KI.ordo helps

In KI.ordo, the “Compare models (benchmark)” template and 👍/👎 ratings of answers support this work; the ratings are used to improve knowledge and instructions, not to train models.

Learn more →: AI Quality Assurance Specialist

Matching templates

Topics

  • Quality
  • Testing
  • Role

Try KI.ordo

Agents, flows and templates in one workspace – free 24-hour demo.

Start demo
← All terms

Related terms

Benchmarking

In AI, benchmarking means systematically comparing models or systems using defined tasks and criteria. Public benchmarks measure things like general knowledge, reasoning or coding ability; for practical use, however, test cases from your own day-to-day work matter most. Typical criteria are accuracy, completeness, response time and cost. This helps decide whether a smaller, cheaper model is good enough for a task.

  • Model comparison
  • Evaluation
  • +1
Template

Hallucination

A hallucination is output from an AI model that sounds convincing but is wrong or entirely made up – such as a non-existent court ruling, an incorrect figure or an invented source. The cause: language models produce likely wording; they do not automatically look up facts. The risk falls when answers are based on verified sources, evidence is cited and people check important results.

  • Language models
  • Reliability
  • +1
Template

Accuracy and stability of AI models

Accuracy describes how often an AI model produces correct results; stability describes how reliably that performance holds up – with slightly altered inputs, new data or over time. A model can be very accurate in testing yet unstable if merely rephrasing a question leads to a different answer. For real-world use, both properties are checked with fixed test cases, cross-validation and ongoing monitoring.

  • Model quality
  • Robustness
  • +1

Model drift

Model drift is the gradual decline in a model’s predictive quality during operation because reality changes compared with the training data. A distinction is made between changes in the input data (data drift) and changes in the relationship between inputs and outcomes (concept drift). Example: a fraud detection model misses new fraud schemes. Remedies include continuous monitoring of metrics and regular retraining.

  • MLOps
  • model quality
  • +1