Lead AI Quality Engineer — Remote

Circle; United States; Remote; USD 190,000 - 230,000; Full time

Benefits offered by this job

Remote-first culture

Equity with refresh grants

35 days PTO per year

Global compensation benchmarking

Job summary

Circle is seeking a Lead Engineer for AI Quality to design evaluation frameworks, observability, and diagnostic tooling for AI agents in a fully remote, globally distributed team. You will build, measure, and optimize how Circle's AI agents perform in production at scale, balancing accuracy, latency, and cost with a hands-on leadership style.

You will manage the AI Quality engineering team, drive experiments across prompts and models, and collaborate with AI Core engineers to ensure meaningful

Qualifications

• 7+ years shipping production software in complex, tool-using systems.
• Experience with LLM-powered agents in products.
• Proficient in Ruby on Rails and Python; RoR core.
• Experience building eval pipelines, dashboards, or CI/CD for evals.
• Experience designing datasets, annotation workflows, or labeling pipelines.
• Thrives in fast-paced, ambiguous environments with experiments.

Responsibilities

• Build and own evaluation infrastructure; CI/CD, scorers, and datasets to validate Circle's AI agents across prompts and multi-turn conversations.
• Diagnose quality breakdowns across the agent pipeline, from planning to execution and tool usage.
• Grow datasets using AI-generated and simulated conversations to broaden coverage.
• Run structured experiments across prompts, models, and routing by plan complexity to improve performance and cost.
• Lead and grow the AI Quality engineering team, setting direction while staying hands-on in code.
• Partner closely with AI Core engineering to ensure changes improve quality, not just ship features.

Job description

Circle is seeking a Lead Engineer for AI Quality to design evaluation frameworks, observability, and diagnostic tooling for AI agents in a fully remote, globally distributed team. You will build, measure, and optimize how Circle's AI agents perform in production at scale, balancing accuracy, latency, and cost with a hands-on leadership style.

You will manage the AI Quality engineering team, drive experiments across prompts and models, and collaborate with AI Core engineers to ensure meaningful