Remote AI Evaluation Engineer: Testing Pipelines

Jobgether; Schweiz; Remote; CHF 83,000 - 165,000; Full time

Benefits offered by this job

Fully remote

Flexible schedule

Professional development

Modern technology

Steady project opportunities

Pay raises

Negotiable compensation

Contractor arrangement

Flexible working environment

Job summary

Jobgether is seeking an AI Evaluation Engineer based in Switzerland for a fully remote contract role. You will design end-to-end evaluation, testing, and observability infrastructure for production-grade AI agents, focusing on reliability, safety, and efficiency.

You will build automated testing systems for complex reasoning and multi-turn conversations, create robust regression/testing methods, and deliver scalable evaluation pipelines.

Qualifications

• Strong Python or TypeScript skills for API testing and automation.
• Hands-on experience building LLM evaluation pipelines or AI testing frameworks.
• Experience integrating tests into CI/CD (GitHub Actions) with SDK-level execution.
• Strong expertise validating JSON schema, OpenAPI specs, and function-calling payloads.
• Familiarity with observability/tracing tools (OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave).
• Ability to design, implement, and maintain an evaluation architecture from scratch.
• Excellent written and spoken English with distributed teams.
• Availability to overlap US Central Time hours daily.
• Experience with code execution sandboxes or AI agent frameworks (LangChain, LlamaIndex, AutoGen).
• Understanding embedding models and vector databases for evaluation.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines before deployment.
• Develop golden datasets for regression prompts and edge cases.
• Implement fast, cost-aware evaluation checks including JSON schema validation.
• Create tests for multi-turn agent conversations and tool-calling loops.
• Design adversarial testing scenarios for prompts and security.
• Build statistical evaluation frameworks with multiple runs and confidence intervals.
• Integrate evaluation workflows into GitHub Actions and similar CI systems.
• Implement observability and tracing to monitor agent execution and diagnose failures.
• Optimize evaluation cost via caching and mocking of LLM calls.
• Validate structured outputs like JSON schemas and OpenAPI specs.

Skills

Python/TypeScript

API testing

CI/CD with GitHub Actions

JSON schema validation

Observability & tracing

LLM tooling (LangChain, LlamaIndex)

English communication

Testing pipelines design

Tools

OpenTelemetry

LangSmith

Arize Phoenix

Weights & Biases Weave (W&B Weave)

Job description

Jobgether is seeking an AI Evaluation Engineer based in Switzerland for a fully remote contract role. You will design end-to-end evaluation, testing, and observability infrastructure for production-grade AI agents, focusing on reliability, safety, and efficiency.

You will build automated testing systems for complex reasoning and multi-turn conversations, create robust regression/testing methods, and deliver scalable evaluation pipelines.