Lead AI Quality Engineer & Platform Observability

CIRCLE; Leinster; Remote; EUR 110,000 - 160,000; Full time

Benefits offered by this job

Remote-first

Biannual company offsites

Competitive compensation

Job summary

Circle is seeking a Lead Engineer for the AI Quality engineering team. You’ll build evaluation infrastructure for Circle's AI agents (planners, tool-callers, sub-agents) and lead a growing team across experimentation and observability, all in a remote-first setting.

You’ll design datasets and annotation workflows, run experiments across models, and collaborate with AI Core to ensure real improvements in reliability, speed, and cost efficiency.

Qualifications

• 7+ years of production software experience, including LLM-powered agents.
• Proficiency in Ruby on Rails and/or Python.
• Experience building evaluation pipelines, dashboards, or CI/CD for evals.
• Experience designing datasets and labeling workflows for ML/AI evaluation.
• Able to learn fast, experiment aggressively, and thrive in a dynamic environment.
• Strong English proficiency (CEFR C2 / ILR 5).

Responsibilities

• Build and own our evaluation infrastructure, design CI/CD pipelines, scorers, and datasets.
• Diagnose where quality breaks down across the agent pipeline and prioritize fixes.
• Grow datasets with annotation workflows and simulated conversations.
• Run structured experiments across prompts, models, and the agent harness.
• Lead and grow the AI Quality engineering team while remaining hands-on.
• Partner with AI Core engineering to ensure changes improve quality.

Skills

LLM-powered agents

Ruby on Rails / Python

Evaluation pipelines

Experimentation

Observability tooling

English proficiency

Tools

Braintrust

LangSmith

CI/CD pipelines

Annotation tooling

Job description

Circle is seeking a Lead Engineer for the AI Quality engineering team. You’ll build evaluation infrastructure for Circle's AI agents (planners, tool-callers, sub-agents) and lead a growing team across experimentation and observability, all in a remote-first setting.

You’ll design datasets and annotation workflows, run experiments across models, and collaborate with AI Core to ensure real improvements in reliability, speed, and cost efficiency.