Lead AI Quality & Platform Engineer

CIRCLE; New York (NY); Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity in the company

35 days paid time off

Remote-only team

Company retreats

Job summary

Circle is seeking a Lead Engineer for the AI Quality team to build evaluation infrastructure, observability tooling, and diagnostic systems for Circle's AI agents. You will lead and grow the team while staying hands-on and partnering with AI Core engineering to improve quality across the product.

This remote, globally distributed role requires experience with LLM-powered products, strong Ruby on Rails and Python skills, and a passion for scalable AI systems.

Qualifications

• 7+ years of experience building and shipping production software, ideally with LLM-powered agents.
• Comfortable in Ruby on Rails / Python or willingness to learn quickly.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Learn fast, experiment aggressively, and thrive in a dynamic environment.
• Comfortable in a fast-paced setting with ambiguity.
• Strong alignment with company values and culture.
• Proficiency in English (CEFR C2 / ILR 5).

Responsibilities

• Build and own our evaluation infrastructure, including CI/CD, scorers, and datasets.
• Diagnose where quality breaks down across the agent pipeline and propose fixes.
• Grow datasets with annotation workflows and AI-generated data.
• Run structured experiments across prompts, models, and the agent harness.
• Lead and grow the AI Quality engineering team while remaining hands-on.
• Collaborate with AI Core engineering to ensure changes improve quality.

Skills

LLM agents

Ruby on Rails

Python

Observability

Evaluation pipelines

Experimentation

English proficiency

Tools

Braintrust

LangSmith

Job description

Circle is seeking a Lead Engineer for the AI Quality team to build evaluation infrastructure, observability tooling, and diagnostic systems for Circle's AI agents. You will lead and grow the team while staying hands-on and partnering with AI Core engineering to improve quality across the product.

This remote, globally distributed role requires experience with LLM-powered products, strong Ruby on Rails and Python skills, and a passion for scalable AI systems.