Remote AI Evaluation Architect

Jobgether; Netherlands; Remote; EUR 49,000 - 123,000; Full time

Benefits offered by this job

Fully remote work

Flexible schedule

Professional development

Modern technology

Independent contractor

Flexible working environment

Job summary

Jobgether, on behalf of our partner, seeks an AI Evaluation Engineer based in the Netherlands to design evaluation, testing, and observability infrastructure for production-grade AI agents. You will build automated testing systems for complex reasoning, multi-turn conversations, and API/tool interactions, enabling reliable and secure AI deployments in a fully remote, globally distributed environment.

You will own the evaluation architecture from scratch, develop regression tests, datasets, and

Qualifications

• Strong proficiency in Python or TypeScript for API testing and automation.
• Experience building LLM evaluation pipelines or AI testing frameworks.
• Experience in CI/CD integration with GitHub Actions or similar.
• Excellent validation of JSON schema, OpenAPI specs, and function-calling payloads.
• Familiarity with observability tools (OpenTelemetry, LangSmith, Arize) and logging.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines for AI agents.
• Create golden datasets and regression prompts to measure accuracy, safety, and latency.
• Implement deterministic evaluation checks and API/tool interactions validation.
• Develop test suites for multi-turn conversations and tool-calling loops.
• Design adversarial tests to probe prompt injections and security weaknesses.
• Build statistical evaluation frameworks with multiple runs and confidence intervals.
• Integrate evaluation workflows into CI systems using SDKs.
• Implement observability and tracing to diagnose failures.

Skills

Python

TypeScript

API testing

CI/CD

Asynchronous programming

JSON schema validation

OpenTelemetry

LangSmith

Arize Phoenix

W&B Weave

LangChain

LlamaIndex

Tools

OpenTelemetry

LangSmith

Arize Phoenix

Weights & Biases Weave

Job description

Jobgether, on behalf of our partner, seeks an AI Evaluation Engineer based in the Netherlands to design evaluation, testing, and observability infrastructure for production-grade AI agents. You will build automated testing systems for complex reasoning, multi-turn conversations, and API/tool interactions, enabling reliable and secure AI deployments in a fully remote, globally distributed environment.

You will own the evaluation architecture from scratch, develop regression tests, datasets, and