AI Quality & Evaluation Engineer for LLM Systems

Koda Tech; Antwerpen; On-site; EUR 60,000 - 90,000; Full time

Job summary

Koda Tech is seeking a QA & AI Evaluation Engineer to shape how we test and monitor a large-scale internal AI platform. This role blends software quality with AI evaluation, focusing on accuracy, safety, and usefulness in production.

You’ll craft AI quality criteria, build golden datasets, and design both deterministic and non-deterministic evaluations. You’ll work across tool calls, RAG, and AI agents while deepening production observability and CI/CD integration.

Qualifications

• Experience as a QA Engineer in an Agile software or IT environment.
• Hands-on testing/evaluation of LLM/Generative AI applications or strong knowledge of AI evaluation methodologies.
• Understanding non-deterministic AI systems and their challenges.

Responsibilities

• Define measurable AI quality and evaluation criteria from business requirements.
• Build and maintain golden datasets with real-world, edge and failure scenarios.
• Design deterministic and non-deterministic evaluations using AI-based approaches.
• Evaluate RAG applications for retrieval quality, groundedness and source accuracy.
• Test AI agents, including tool selection, parameters, multi-turn behavior and error handling.
• Carry out adversarial testing and AI red teaming (prompt injection, jailbreaks, data leakage).
• Test and validate guardrails and content filters.
• Monitor regression after model upgrades and prompt changes.
• Integrate automated AI evaluations into CI/CD and delivery pipelines.
• Monitor AI quality in production using observability and tracing.
• Support UAT sessions and turn production feedback into repeatable test cases.

Skills

QA Engineer

AI quality evaluation

Non-deterministic AI testing

Dutch

English

Tools

Python / pytest

LangSmith

Langfuse

promptfoo

DeepEval

Ragas

Azure AI Foundry Evaluations

RAG & embeddings

LLM evaluation

AI agents & tool calling

API testing

REST

Postman

OpenAPI

SQL

JSON

OpenTelemetry

Application Insights

Job description

Koda Tech is seeking a QA & AI Evaluation Engineer to shape how we test and monitor a large-scale internal AI platform. This role blends software quality with AI evaluation, focusing on accuracy, safety, and usefulness in production.

You’ll craft AI quality criteria, build golden datasets, and design both deterministic and non-deterministic evaluations. You’ll work across tool calls, RAG, and AI agents while deepening production observability and CI/CD integration.