Remote Lead Engineer - AI Quality & Observability

AI Chopping Block, Inc.; Amsterdam; Remote; EUR 120,000 - 180,000; Full time

Benefits offered by this job

Remote-first

Global offsites

Equity grants

Job summary

Circle is a fully remote, global team building an AI-powered, all-in-one platform for creators and businesses. We are seeking a Lead Engineer for the AI Quality team to own evaluation frameworks, observability tooling, and diagnostic infrastructure.

This hands-on leader will mentor the team and partner with AI Core to ensure our AI Agents perform reliably, efficiently, and at scale. You will design evaluators, run experiments, and grow data pipelines for better evaluation coverage.

Qualifications

• 7+ years of experience shipping production software with complex, tool-using systems.
• Proficiency or quick learning in Ruby on Rails and Python.
• Experience building eval pipelines, dashboards or CI/CD for ML/AI evals.
• Experience designing datasets and labeling pipelines for ML/AI evaluation.
• Ability to run experiments and derive insights from results.

Responsibilities

• Build and own evaluation infrastructure, including pipelines and datasets.
• Diagnose quality issues across the agent workflow and prioritize fixes.
• Grow annotation workflows and AI-generated datasets for evaluation.
• Run structured experiments across prompts, models, and agent harnesses.
• Lead and grow the AI Quality engineering team, while staying hands-on.
• Collaborate with AI Core engineering to improve product quality.

Skills

LLM-powered agents

Ruby on Rails

Python

Experimentation

Leadership

Tools

Braintrust

LangSmith

Job description

Circle is a fully remote, global team building an AI-powered, all-in-one platform for creators and businesses. We are seeking a Lead Engineer for the AI Quality team to own evaluation frameworks, observability tooling, and diagnostic infrastructure.

This hands-on leader will mentor the team and partner with AI Core to ensure our AI Agents perform reliably, efficiently, and at scale. You will design evaluators, run experiments, and grow data pipelines for better evaluation coverage.