AI Quality & Evaluation Architect for LLM Systems

DutchTech; España; Hybrid; PHP 4,291,845 - 6,131,208; Full time

Benefits offered by this job

Stock options

Work & Swim program in Cyprus

Flexible work model across Europe

Job summary

DutchTech in the Philippines seeks a specialist in AI safety who will manage the quality and performance of AI systems. Responsibilities include designing automated evaluation frameworks, curating benchmarks, and ensuring high-quality outputs across multi-agent workflows.

The ideal candidate should possess Python coding skills, analytical thinking for testing AI systems, and have a collaborative mindset. Additional perks include remote work flexibility and opportunities for stock options.

Qualifications

• Strong Python coding and software engineering fundamentals.
• Deep understanding of LLMs and analytical testing mindset.
• Experience coordinating across technical teams.

Responsibilities

• Design and maintain scalable evaluation frameworks.
• Collaborate to build and sanitize gold-standard datasets.
• Monitor and enforce quality KPIs for agent workflows.
• Lead analyses to mitigate systemic risks in outputs.
• Integrate automated testing gates with engineering teams.
• Analyze efficiency of models for optimization.

Skills

Python coding experience

Software engineering fundamentals

Analytical thinking

Understanding of token behaviours

Tools

LangSmith

DeepEval

Ragas

Opik

Job description

DutchTech in the Philippines seeks a specialist in AI safety who will manage the quality and performance of AI systems. Responsibilities include designing automated evaluation frameworks, curating benchmarks, and ensuring high-quality outputs across multi-agent workflows.

The ideal candidate should possess Python coding skills, analytical thinking for testing AI systems, and have a collaborative mindset. Additional perks include remote work flexibility and opportunities for stock options.