Remote AI Evaluation Architect

Jobgether; France; Remote; EUR 83,000 - 138,000; Full time

Benefits offered by this job

Fully remote work

Flexible schedule

Professional development

Modern technology

Independent contractor arrangement

Negotiable compensation

Job summary

Jobgether is seeking a founding AI Evaluation Engineer based in France to design and implement scalable evaluation pipelines for autonomous AI agents. You will build automated testing, regression coverage, and adversarial scenarios, enabling reliable, secure deployments across a distributed, fully remote team.

You will own the evaluation architecture, integrate with CI/CD, implement observability, and drive cost-efficient testing practices, working with modern AI tools and frameworks.

Qualifications

• Strong proficiency in Python or TypeScript for API testing and automation.
• Experience building LLM evaluation pipelines and AI testing frameworks.
• Experience integrating tests into CI/CD, esp. GitHub Actions.
• Expertise validating JSON schemas, OpenAPI specs, and function-calling payloads.
• Familiarity with observability/tracing tools like OpenTelemetry and LangSmith.
• Ability to design and own an evaluation architecture from scratch.
• Excellent written and spoken English with global collaboration.

Responsibilities

• Design, build, and maintain automated CI/CD testing pipelines for AI agents.
• Develop golden datasets for regression prompts and edge cases.
• Implement deterministic evaluation checks (JSON validation, API status, embeddings).
• Create test suites for multi-turn agent conversations.
• Design adversarial testing scenarios for prompt injection and vulnerabilities.
• Build statistical evaluation frameworks for non-deterministic behavior.
• Integrate evaluation workflows into GitHub Actions via SDK execution.
• Implement observability and tracing to monitor agent execution.
• Optimize evaluation cost via caching and mocking.
• Validate structured outputs (JSON schemas, OpenAPI, function calls).
• Evaluate performance under load and varying responses.
• Own the evaluation architecture and establish best practices.

Skills

Python/TypeScript

LLM evaluation pipelines

CI/CD

JSON schema/OpenAPI validation

Observability tools

LangChain/LLM frameworks

English communication

US time overlap

Tools

OpenTelemetry

LangSmith

Arize Phoenix

Weights & Biases Weave

Job description

Jobgether is seeking a founding AI Evaluation Engineer based in France to design and implement scalable evaluation pipelines for autonomous AI agents. You will build automated testing, regression coverage, and adversarial scenarios, enabling reliable, secure deployments across a distributed, fully remote team.

You will own the evaluation architecture, integrate with CI/CD, implement observability, and drive cost-efficient testing practices, working with modern AI tools and frameworks.