Lead AI Quality Engineer — Evaluation & Scale

AI Chopping Block, Inc.; Dublin; Remote; EUR 140,000 - 190,000; Full time

Benefits offered by this job

Remote‑only team

35 days PTO

Equity with refresh grants

Job summary

Circle is seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks, observability tooling and diagnostic infrastructure. You will lead a cross-functional team and stay hands-on in code while guiding a growing roadmap.

You will evaluate AI agents, plan experiments, and work with AI Core engineering to ensure reliability, speed, and cost efficiency across production systems in a remote, globally distributed environment.

Qualifications

• 7+ years shipping production software, incl. LLM-powered agents.
• Experience building eval pipelines, dashboards or CI/CD for evals.
• Experience designing datasets, annotation workflows, or labeling pipelines.

Responsibilities

• Build and own evaluation infrastructure; design pipelines, scorers and datasets for agent evaluation.
• Diagnose failures across the agent pipeline and prioritise fixes for AI core engineering.
• Grow annotation workflows and AI-generated/simulated conversations to expand coverage.
• Run structured experiments across prompts, models, and the agent harness to reduce cost and latency.
• Lead and grow the AI Quality engineering team while staying hands-on in code.
• Collaborate with AI Core engineering to improve product quality and confidence.

Skills

Experience with production software

Ruby on Rails

Python

ML/AI eval infra

Dataset design & annotation

Experimentation mindset

English proficiency (C2/ILR5)

Tools

Braintrust

LangSmith

Job description

Circle is seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks, observability tooling and diagnostic infrastructure. You will lead a cross-functional team and stay hands-on in code while guiding a growing roadmap.

You will evaluate AI agents, plan experiments, and work with AI Core engineering to ensure reliability, speed, and cost efficiency across production systems in a remote, globally distributed environment.