Lead AI Quality Engineer – Remote & Impactful

AI Chopping Block, Inc.; Warszawa; Hybrid; PLN 583,000 - 894,000; Full time

Benefits offered by this job

Equity

35 days PTO

Job summary

Circle is seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks, observability tooling, and diagnostic infrastructure for Circle's AI agents. This hands-on player-coach role will lead the team while remaining technically active.

You'll collaborate with AI Core, design datasets, run experiments across models and prompts, and drive cost and latency improvements. If you thrive in a fast-paced, ambiguous environment and value reliability at scale, this role is

Qualifications

• 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
• Comfortable in Ruby on Rails / Python or ready to pick them up quickly.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Learn fast, experiment aggressively, and thrive in a highly dynamic environment.
• Comfortable in a fast-paced environment with ambiguity.
• Strong alignment with our company values.
• You are proficient in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.

Responsibilities

• Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents are actually working, from a single tool call to a full multi-turn conversation.
• Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics.
• Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone.
• Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity.
• Lead and grow the AI Quality engineering team. Set technical direction and manage day-time priorities, all while staying hands-on in the code yourself.
• Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.

Skills

7+ years exp

Ruby on Rails

Python

ML evaluation infra

Experimentation

English proficiency

Tools

Braintrust

LangSmith

Job description

Circle is seeking a Lead Engineer for the AI Quality engineering team to build evaluation frameworks, observability tooling, and diagnostic infrastructure for Circle's AI agents. This hands-on player-coach role will lead the team while remaining technically active.

You'll collaborate with AI Core, design datasets, run experiments across models and prompts, and drive cost and latency improvements. If you thrive in a fast-paced, ambiguous environment and value reliability at scale, this role is