Remote Lead AI Quality & Evaluation Engineer

CIRCLE; Amsterdam; Remote; EUR 133,000 - 169,000; Full time

Benefits offered by this job

Equity grants

35 days paid time off

Job summary

Circle is seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks and observability tooling for AI-powered features. You will lead a hands-on team, define the technical direction, and drive improvements across the agent pipeline.

In this remote, globally distributed role, you will own datasets, run experiments, and collaborate with AI Core engineering to ensure reliability, speed, and cost efficiency of Circle's AI agents at scale.

Qualifications

• 7+ years of experience building and shipping production software.
• Experience with LLM-powered agents that take real actions in a product.
• Proficiency in Ruby on Rails and/or Python.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Learn fast, experiment aggressively in a dynamic environment.
• Strong English communication at CEFR C2/ILR 5.

Responsibilities

• Build and own evaluation infrastructure, CI/CD pipelines, scorers, and datasets.
• Diagnose where quality breaks down across the agent pipeline.
• Grow annotation workflows and AI-generated or simulated conversations.
• Run structured experiments across prompts, models, and the agent harness.
• Lead and grow the AI Quality engineering team while staying hands-on.
• Partner with AI Core engineering to improve quality.

Skills

LLM-powered agents

Production software

Observability

Experimentation

Team leadership

Tools

Ruby on Rails

Python

Braintrust

LangSmith

Job description

Circle is seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks and observability tooling for AI-powered features. You will lead a hands-on team, define the technical direction, and drive improvements across the agent pipeline.

In this remote, globally distributed role, you will own datasets, run experiments, and collaborate with AI Core engineering to ensure reliability, speed, and cost efficiency of Circle's AI agents at scale.