Remote Lead: AI Platform & Quality Engineering

Jobgether SRL; France; Remote; EUR 129,000 - 174,000; Full time

Benefits offered by this job

Fully remote work environment

35 days paid time off per year

Equity and refresh grants

Global team collaboration

Job summary

Jobgether SRL, on behalf of a partner company, seeks a Lead Engineer, AI Platform based in France to lead the engineering foundation behind reliable, scalable AI-powered features. You will build evaluation frameworks, observability tooling, and diagnostic infrastructure for production AI systems.

This role blends hands-on software development with technical leadership, dataset development, experimentation, and cost/latency optimization in a highly remote, asynchronous environment.

Qualifications

• 7+ years of experience building and shipping production software, ideally including LLM-powered agents.
• Experience with complex, tool-using AI systems involving multiple tools, planning, orchestration, or sub-agents.
• Strong ability to demonstrate shipped software and explain how its effectiveness and reliability were measured.
• Experience with Ruby on Rails and/or Python, with ability to become productive quickly in new technologies.
• Experience building evaluation or observability infrastructure for ML/AI systems, including evaluation pipelines, scorers, dashboards, or CI/CD systems for evaluations.
• Familiarity with evaluation frameworks such as Braintrust, LangSmith, or similar tools.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Ability to learn quickly, experiment extensively, and use empirical results to guide technical decisions.
• Comfortable operating in a fast-paced environment with ambiguity and changing technical requirements.
• Strong technical leadership and people-management capabilities.

Responsibilities

• Design, build, and own evaluation infrastructure, including CI/CD pipelines, scorers, datasets, and systems for assessing AI agents from individual tool calls through complete multi-turn conversations.
• Develop observability and diagnostic capabilities to identify exactly where quality issues occur across planning, execution, tool selection, and complex agent trajectories.
• Investigate failures across sophisticated AI workflows and turn findings into technical prototypes, improvements, or clearly defined priorities for AI engineering teams.
• Build and expand datasets through human annotation, AI-generated examples, and simulated conversations to increase evaluation coverage efficiently.
• Develop structured experimentation frameworks for prompts, models, and agent harnesses, including evaluation of new and open-source models against production baselines.
• Identify opportunities to improve AI system cost and latency through model selection, caching, routing, and other optimization strategies.
• Set the technical direction and manage day-to-day priorities for the AI Quality engineering team while remaining actively involved in hands-on development.
• Partner closely with AI Core engineering teams to ensure changes to AI products can be measured effectively and demonstrably improve quality.
• Establish engineering practices and evaluation approaches that support reliable, efficient, and scalable production AI systems.

Skills

LLM-powered agents

Observability

Experimentation

Leadership

People management

English proficiency

Tools

Ruby on Rails

Python

CI/CD pipelines

Braintrust LangSmith

Job description

Jobgether SRL, on behalf of a partner company, seeks a Lead Engineer, AI Platform based in France to lead the engineering foundation behind reliable, scalable AI-powered features. You will build evaluation frameworks, observability tooling, and diagnostic infrastructure for production AI systems.

This role blends hands-on software development with technical leadership, dataset development, experimentation, and cost/latency optimization in a highly remote, asynchronous environment.