Remote AI Evaluation Engineer: Reliable AI Testing

Jobgether; Belgique; Remote; EUR 83,000 - 152,000; Full time

Benefits offered by this job

Fully remote

Flexible schedule

Professional development

Modern technology

Independent contractor

Negotiable compensation

Global team

Job summary

Jobgether is seeking a founding AI Evaluation Engineer based in Belgium. You will design and build scalable evaluation pipelines for AI agents, ensuring safety, reliability, and efficiency across multi-turn conversations and API tool interactions.

This is a fully remote, globally distributed opportunity requiring strong Python/TypeScript skills and CI/CD experience. You will establish robust testing, regression suites, and adversarial scenarios, while integrating workflows into GitHub Actions

Qualifications

• Strong proficiency in Python or TypeScript for API testing and automation.
• Experience building LLM evaluation pipelines or AI testing frameworks.
• Experience integrating automated tests into CI/CD environments, preferably GitHub Actions.
• Strong ability to validate structured data including JSON schemas, OpenAPI specs, and function-calling payloads.
• Familiarity with observability/tracing tools such as OpenTelemetry and LangSmith.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines for AI agents before deployment and throughout development lifecycle.
• Develop golden datasets with regression prompts, edge cases, and synthetic scenarios to measure accuracy, safety, performance, and latency.
• Implement fast, deterministic evaluation checks including JSON schema validation and API status checks.
• Create comprehensive test suites for multi-turn agent conversations, context retention, and safeguards against loops.
• Design adversarial testing scenarios to assess resilience against prompt injection and malformed inputs.
• Build statistical evaluation frameworks with pass rates across runs and confidence intervals.
• Integrate evaluation workflows into GitHub Actions or similar CI systems.
• Implement observability and tracing to monitor agent execution and diagnose failures.
• Optimize evaluation cost with caching and reduced LLM API calls.
• Validate outputs against JSON schemas, OpenAPI specs, and function-calling payloads.
• Evaluate performance under load and non-deterministic endpoints.
• Independently design and own the evaluation architecture and processes.

Skills

Python/TypeScript

API testing

Asynchronous

Automated testing

CI/CD

JSON schema

Observability

English comms

Tools

GitHub Actions

OpenTelemetry

LangSmith

Arize Phoenix

Weave by W&B

LangChain

LlamaIndex

AutoGen

CrewAI

Job description

Jobgether is seeking a founding AI Evaluation Engineer based in Belgium. You will design and build scalable evaluation pipelines for AI agents, ensuring safety, reliability, and efficiency across multi-turn conversations and API tool interactions.

This is a fully remote, globally distributed opportunity requiring strong Python/TypeScript skills and CI/CD experience. You will establish robust testing, regression suites, and adversarial scenarios, while integrating workflows into GitHub Actions