Lead AI Quality Engineer — Observability & Evaluation

Circle; Northern (KY); Hybrid; USD 150,000 - 190,000; Full time

Benefits offered by this job

Equity

Remote-friendly

35 days PTO

Job summary

Circle is seeking a Lead Engineer for AI Quality to own and evolve the evaluation frameworks, observability tooling, and diagnostic infrastructure for Circle's AI-powered features. You will lead a growing team while remaining hands-on in code, ensuring reliable, efficient AI systems at scale.

The role emphasizes building evaluation pipelines, datasets, and CI/CD workflows, with collaboration across AI core engineering. Strong Ruby on Rails or Python skills and experience in ML evals are required.

Qualifications

• 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
• Comfortable in Ruby on Rails / Python or ready to pick them up quickly.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Learn fast, experiment aggressively, and thrive in a highly dynamic environment.
• Comfortable in a fast-paced environment with ambiguity.
• Strong alignment with our company values. You are proficient in English (CEFR Level C2 / ILR Level 5).

Responsibilities

• Build and own our evaluation infrastructure. Design CI/CD pipelines, scorers, and datasets that determine AI agents performance.
• Diagnose exactly where quality breaks down across the agent pipeline.
• Grow the datasets with AI-generated and simulated conversations for faster coverage.
• Run structured experiments across prompts, models, and the agent harness; evaluate models and optimize costs.
• Lead and grow the AI Quality engineering team; set technical direction while staying hands-on.
• Partner with AI Core engineering to ensure changes improve quality, not just ship features.

Skills

7+ years experience

Ruby on Rails

Python

Evaluation/observability infra

Dataset annotation workflows

Experimentation across prompts/models

Fast learner / adaptable

English proficiency (C2/ILR5)

Tools

Braintrust

LangSmith

Job description

Circle is seeking a Lead Engineer for AI Quality to own and evolve the evaluation frameworks, observability tooling, and diagnostic infrastructure for Circle's AI-powered features. You will lead a growing team while remaining hands-on in code, ensuring reliable, efficient AI systems at scale.

The role emphasizes building evaluation pipelines, datasets, and CI/CD workflows, with collaboration across AI core engineering. Strong Ruby on Rails or Python skills and experience in ML evals are required.