Lead AI Quality Engineer — Remote

Circle; Oregon; Hybrid; USD 180,000 - 260,000; Full time

Benefits offered by this job

Remote-first company

Equity in the company

35 days paid time off

Job summary

Circle’s AI Quality engineering team is seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for AI agents. This hands-on player-coach role also leads the team with an ambitious roadmap.

You’ll partner with AI Core engineering, design eval pipelines, and drive experiments to improve quality, speed, and cost efficiency of Circle’s AI features.

Qualifications

• 7+ years of experience building and shipping production software with complex, tool-using systems.
• Experience with evaluation or observability infrastructure for ML/AI systems.
• Experience building eval pipelines, scorers, dashboards, or CI/CD for evals.
• Proficiency in Ruby on Rails and/or Python.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Ability to run many experiments and let results determine decisions.

Responsibilities

• Build and own evaluation infrastructure for Circle's AI agents and workflows.
• Design CI/CD pipelines, scorers, and datasets to validate agent performance.
• Diagnose quality breakdowns across plan creation, execution, and tool usage.
• Develop annotation workflows and synthetic data to accelerate testing.
• Run structured experiments across prompts, models, and agent harnesses.
• Evaluate new/open-source models against production baselines and manage model swaps.
• Lead and grow the AI Quality engineering team while staying hands-on in code.
• Collaborate closely with AI Core engineering and product teams.

Skills

LLM-powered agents

Evaluation pipelines

Observability infrastructure

Ruby on Rails

Python

CI/CD for evals

Dataset design

Tools

Braintrust

LangSmith

Job description

Circle’s AI Quality engineering team is seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for AI agents. This hands-on player-coach role also leads the team with an ambitious roadmap.

You’ll partner with AI Core engineering, design eval pipelines, and drive experiments to improve quality, speed, and cost efficiency of Circle’s AI features.