Lead AI Quality Engineer - Remote, Scale & Impact

Circle; Netherlands; Remote; EUR 120,000 - 180,000; Full time

Benefits offered by this job

35 days paid time off

Remote-only team

Equity in the company

Job summary

Circle is seeking a Lead Engineer for AI Quality to build evaluation frameworks and observability tooling for AI-powered agents. This role is hands-on, leading the team while contributing code, and requires deep experience with production ML/AI systems.

You will partner with AI Core engineering to ensure improvements in quality, latency, and cost are achieved across a global, remote-first team.

Qualifications

• 7+ years of experience building and shipping production software, including LLM-powered agents.
• Strong proficiency in Ruby on Rails and Python; production systems.
• Experience building evaluation or observability infrastructure for ML/AI systems.

Responsibilities

• Build and own our evaluation infrastructure, including CI/CD pipelines, scorers, and datasets.
• Diagnose where quality breaks down across the agent pipeline and propose solutions.
• Grow datasets with annotation workflows and AI-generated or simulated conversations.
• Run structured experiments across prompts, models, and agent harness; evaluate models against production baseline.
• Lead and grow the AI Quality engineering team, setting direction while remaining hands-on.
• Collaborate closely with AI Core engineering to ensure changes improve quality.

Skills

English proficiency

Ruby on Rails

Python

Tools

Ruby on Rails

Python

Job description

Circle is seeking a Lead Engineer for AI Quality to build evaluation frameworks and observability tooling for AI-powered agents. This role is hands-on, leading the team while contributing code, and requires deep experience with production ML/AI systems.

You will partner with AI Core engineering to ensure improvements in quality, latency, and cost are achieved across a global, remote-first team.