QA & AI Evaluation Engineer - Antwerp office

Delen Private Bank; Antwerpen; On-site; EUR 60,000 - 80,000; Full time

Benefits offered by this job

Hospitalisation insurance

Dental insurance

Group insurance

Mobility budget

Job summary

Delen Private Bank is seeking a QA & AI Evaluation Engineer to safeguard the quality, reliability, and safety of the AIHub Platform. You will define evaluation criteria, build datasets, and automate evaluations integrated into delivery pipelines.

Collaboration with AI engineers, developers, analysts and stakeholders is essential to translate business expectations into measurable criteria. You will design evaluation strategies, run tests on APIs, data processing and UIs, and monitor model

Qualifications

• Experience as a QA Engineer in an agile IT environment
• Hands-on testing or evaluation of LLM applications
• Ability to translate business expectations into measurable evaluation criteria
• Familiarity with non-deterministic AI systems and edge cases
• Strong analytical and structured thinking
• Excellent communication across technical and non-technical stakeholders

Responsibilities

• Translate AI behaviour expectations into measurable acceptance criteria
• Define evaluation datasets including edge cases and failure scenarios
• Develop deterministic checks: format validation, schema checks, citations, correct tool calls
• Design and calibrate LLM-as-a-judge evaluations against human judgments
• Run functional and non-functional tests on platform components
• Monitor quality in production and track regression after model upgrades
• Write annotation guidelines and review labelling outputs
• Communicate findings clearly to stakeholders

Skills

QA engineering

AI evaluation

Analytical thinking

Team collaboration

Tools

Python

LLM evaluation frameworks

Postman/OpenAPI

Azure DevOps

SQL/JSON

Job description

Delen Private Bank is a private wealth manager with a well-established IT team of 200 colleagues within an organisation of 800 employees. We support our clients through discretionary wealth management and wealth planning, and are strongly committed to automation through our strategically developed in-house core banking software.

Role and context

As QA & AI Evaluation Engineer, you safeguard the quality, reliability and safety of the AIHub Platform. This is Delen Private Bank’s central in-house AI platform, integrated into the bank’s processes and systems.

AI solutions do not always behave predictably. That is why quality here goes beyond "does the functionality work as intended?" You define how we measure whether an AI application is correct, consistent, safe and useful. You also check that it stays that way after changes to models, prompts or data.

You work closely with AI engineers, developers, business analysts, domain experts and stakeholders. You combine strong QA expertise with analytical insight into AI systems. You translate business expectations into measurable evaluation criteria. You build and maintain evaluation datasets and embed automated evaluations in our delivery pipelines.

Responsibilities

Analysis & Evaluation Design

• Work with business analysts and domain experts to translate expectations about AI behaviour into measurable acceptance and evaluation criteria.
• Define evaluation datasets (golden datasets) that represent real-world usage scenarios. Include edge cases and failure scenarios.
• deterministic: e.g. format validation, schema checks, exact or structured matches, mandatory source citation, correct tool calls;
• non-deterministic: e.g. LLM-as-a-judge, semantic similarity, groundedness and relevance.
• Calibrate LLM-as-a-judge evaluations against human judgements, so that automated scores stay reliable.
• Help shape the evaluation strategy, coverage and quality thresholds per use case and for the platform as a whole.

Test Execution & Validation

• Run functional and non-functional tests on platform components: APIs, integrations, data processing and user interfaces.
• Evaluate RAG applications for retrieval quality, groundedness and correct source citation.
• Test agentic flows: correct tool selection, parameters, multi-turn behaviour and error handling.
• Perform adversarial testing and red teaming: prompt injection, jailbreaks, leakage of confidential or personal data, unwanted or misleading output.
• Validate that guardrails and content filters work.
• Log bugs in Azure DevOps, actively follow up on their resolution and validate fixed issues.
• Facilitate UAT sessions with end users and collect structured feedback.
• Monitor regression on model upgrades and prompt changes, and compare versions objectively.
• Maintain evaluation datasets based on online evaluations and production feedback. You turn unforeseen edge cases into permanent test cases within the standard eval sets.
• Track quality in production through monitoring and tracing (quality scores, drift, user feedback) and flag degradation early.
• Use AI tooling yourself to generate test cases and synthetic test data and to analyse results faster.
• Help build reusable evaluation standards, templates and guidelines, so that teams building on AIHub can measure the quality of their own use cases.
• Write annotation guidelines and involve domain experts in reviewing and labelling output.
• Communicate findings clearly to technical and non-technical stakeholders.

Profile and competencies

• Excellent Dutch and good English. French is a plus (multilingual evaluation).
• Experience as a QA Engineer in an agile IT environment, with strong analytical skills.
• You have hands-on experience testing or evaluating LLM applications, or you show clear maturity in this domain.
• You understand the challenges of non-deterministic systems and report on them reliably and with statistical rigour.
• You have a critical and creative "break it" mindset to find edge cases and abuse scenarios.
• Accurate, structured and quality-driven.
• Team player with excellent communication skills.

Basic knowledge of or experience with:

• Python for test automation and evaluation scripts (e.g. pytest)
• LLM evaluation frameworks (e.g. LangSmith, Azure AI Foundry evaluations, promptfoo, DeepEval, Ragas, Langfuse)
• Concepts such as RAG, embeddings, prompt engineering, agents and tool calling
• API testing (e.g. Postman, REST, OpenAPI)
• Azure DevOps for test management and CI/CD
• SQL / JSON for validating data and interfaces
• Observability tooling (e.g. OpenTelemetry, Application Insights)
• Experience with or affinity for the (private) banking sector
• Agile ways of working (Scrum/Kanban)

We offer

• Attractive conditions: you will receive a competitive salary supplemented with an extensive package of fringe benefits, such as hospitalisation and dental insurance, group insurance, eco vouchers, mobile phone with subscription, option plan, extra days off (bank holidays), Benefits at Work, restaurant with freshly prepared meals, choice of company car or other (mobility) solutions via the mobility budget.
• Development opportunities & intrapreneurship: your growth is central. We listen to your ambitions and actively help you to take steps within our organization. Through training, internal mobility and intrapreneurship, you will be given the space to develop yourself and take on new challenges. You are the architect of your career – we provide the right support.
• Personal approach: as a family business since 1936, we believe in the power of a personal approach. You work in a warm and involved environment where you are really seen. Thanks to our flat structure, the lines of communication are short and the management – including the family – is accessible and involved. Ideas are heard quickly, decisions are made decisively.
• Collegial atmosphere: collaboration is in our DNA. You will be part of a close-knit team where colleagues support each other, share knowledge and motivate each other. We work on shared goals in an open and accessible atmosphere – with success and with room for humour.
• Sustainable collaboration: we build long-term relationships – with our customers and our employees. Our very low staff turnover is proof of this. Thanks to a culture of trust, open communication and stability, you will have the opportunity to grow with us.