Remote Lead Engineer: AI Quality & Evaluation

Circle; United States; Remoto; USD 180.000 - 230.000; Tempo pieno

Vantaggi offerti da questo lavoro

Equity with refresh grants

35 days paid time off

Remote‑only team

Company retreats twice a year

Health and wellbeing benefits

Descrizione del lavoro

Circle is a remote-first company building the world’s leading AI-powered platform for digital businesses. We’re seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks, observability tooling, and diagnostics for Circle’s AI agents, guiding multi-turn conversations with a hands-on, player-coach approach.

You will lead and grow the team, set technical direction, and collaborate closely with AI Core engineering to ensure meaningful quality improvements across

Competenze

• 7+ years of building and shipping production software with AI-enabled components.
• Experience with LLM-powered agents and multi-tool orchestration.
• Strong Ruby on Rails / Python proficiency and fast learning.
• Experience building eval pipelines, dashboards, or CI/CD for evals.
• Ability to design datasets and labeling workflows for ML/AI evaluation.
• Comfort with experiments and ambiguity; strong communication.

Mansioni

• Build and own our evaluation infrastructure, including CI/CD, scorers, and datasets.
• Diagnose quality breakdowns across agent pipelines and prioritize improvements.
• Grow annotation workflows and AI-generated/conversation datasets.
• Run structured experiments across prompts, models, and routing by plan complexity.
• Lead and grow the AI Quality engineering team, setting direction while staying hands-on.
• Partner with AI Core engineering to ensure changes improve quality, not just shipping.

Conoscenze

LLM-powered agents

Ruby on Rails

Python

Evaluation pipelines

Observability tooling

Experimentation

CI/CD

Dataset construction

Leadership

English proficiency

Strumenti

Braintrust

LangSmith

Descrizione del lavoro

Circle is a remote-first company building the world’s leading AI-powered platform for digital businesses. We’re seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks, observability tooling, and diagnostics for Circle’s AI agents, guiding multi-turn conversations with a hands-on, player-coach approach.

You will lead and grow the team, set technical direction, and collaborate closely with AI Core engineering to ensure meaningful quality improvements across