QA & AI Evaluation Engineer

Koda Tech; Antwerpen; On-site; EUR 60,000 - 90,000; Full time

Job summary

Koda Tech is seeking a QA & AI Evaluation Engineer to shape how we test and monitor a large-scale internal AI platform. This role blends software quality with AI evaluation, focusing on accuracy, safety, and usefulness in production.

You’ll craft AI quality criteria, build golden datasets, and design both deterministic and non-deterministic evaluations. You’ll work across tool calls, RAG, and AI agents while deepening production observability and CI/CD integration.

Qualifications

• Experience as a QA Engineer in an Agile software or IT environment.
• Hands-on testing/evaluation of LLM/Generative AI applications or strong knowledge of AI evaluation methodologies.
• Understanding non-deterministic AI systems and their challenges.

Responsibilities

• Define measurable AI quality and evaluation criteria from business requirements.
• Build and maintain golden datasets with real-world, edge and failure scenarios.
• Design deterministic and non-deterministic evaluations using AI-based approaches.
• Evaluate RAG applications for retrieval quality, groundedness and source accuracy.
• Test AI agents, including tool selection, parameters, multi-turn behavior and error handling.
• Carry out adversarial testing and AI red teaming (prompt injection, jailbreaks, data leakage).
• Test and validate guardrails and content filters.
• Monitor regression after model upgrades and prompt changes.
• Integrate automated AI evaluations into CI/CD and delivery pipelines.
• Monitor AI quality in production using observability and tracing.
• Support UAT sessions and turn production feedback into repeatable test cases.

Skills

QA Engineer

AI quality evaluation

Non-deterministic AI testing

Dutch

English

Tools

Python / pytest

LangSmith

Langfuse

promptfoo

DeepEval

Ragas

Azure AI Foundry Evaluations

RAG & embeddings

LLM evaluation

AI agents & tool calling

API testing

REST

Postman

OpenAPI

SQL

JSON

OpenTelemetry

Application Insights

Job description

We’re looking for a QA & AI Evaluation Engineer to join a team building and improving a large-scale internal AI platform.

This isn’t a traditional QA role.

You’ll be working at the intersection of software quality and AI, helping define how Generative AI applications are tested, evaluated and monitored in real-world production environments.

As LLM-based applications aren’t always deterministic, you’ll help answer a much more interesting question than simply "does it work?":

Is the AI accurate, consistent, safe and genuinely useful?

What you’ll be doing

• Define measurable AI quality and evaluation criteria based on business requirements.
• Build and maintain golden datasets covering real-world scenarios, edge cases and failure scenarios.
• Design deterministic and non-deterministic evaluations using techniques such as:
• LLM-as-a-judge
• Semantic similarity
• Groundedness and relevance
• Schema and format validation
• Source/citation validation
• Tool-call validation
• Evaluate RAG applications for retrieval quality, groundedness and source accuracy.
• Test AI agents, including tool selection, parameters, multi-turn behaviour and error handling.
• Carry out adversarial testing and AI red teaming, including prompt injection, jailbreaks, data leakage and misleading outputs.
• Test and validate guardrails and content filters.
• Monitor regression following model upgrades and prompt changes.
• Integrate automated AI evaluations into CI/CD and delivery pipelines.
• Monitor AI quality in production using observability and tracing.
• Support UAT sessions and turn production feedback into repeatable test cases.

What we’re looking for

• Experience working as a QA Engineer within an Agile software or IT environment.
• Hands-on experience testing or evaluating LLM / Generative AI applications, or strong knowledge of AI evaluation methodologies.
• Good understanding of the challenges involved in testing non-deterministic AI systems.
• Strong analytical and problem-solving skills.
• A naturally critical mindset with an interest in finding edge cases, unexpected behaviours and potential failure scenarios.
• Excellent Dutch and good English. French is a plus.
• Comfortable working across both technical teams and business stakeholders.

Technical experience

You don’t need to tick every box, but experience with some of the following would be highly relevant:

• Python / pytest
• LangSmith, Azure AI Foundry Evaluations, promptfoo, DeepEval, Ragas or Langfuse
• RAG & embeddings
• LLM evaluation
• AI agents & tool calling
• API testing: REST, Postman, OpenAPI
• SQL / JSON
• OpenTelemetry / Application Insights

Why consider it?

This is a good opportunity for someone who already has a strong QA/testing foundation but wants to specialise further in AI quality and evaluation.

You’ll get hands‑on exposure to LLMs, RAG, AI agents, automated evaluation, red teaming and production monitoring, while helping establish the standards used to determine whether AI applications are genuinely ready for production.

If your background sits somewhere between QA Engineering, Test Automation and AI/LLM evaluation, I’d be keen to speak.