Lead AI Quality Engineer - Remote & Observability

Circle; Polska; Remote; PLN 544,000 - 816,000; Full time

Benefits offered by this job

Remote‑only team

Global compensation

35 days PTO

Job summary

Circle is a fully remote, global team building AI-powered, all-in-one platforms for creators and businesses. We seek a Lead Engineer to own evaluation infrastructure, diagnose quality in AI agents, and lead an expanding AI Quality engineering team while staying hands-on in code.

The role emphasizes building eval pipelines, datasets, experiments, and collaborating with AI Core to improve reliability, speed, and cost efficiency across production systems.

Qualifications

• 7+ years of experience building and shipping production software for complex, tool‑using systems.
• Proficiency in Ruby on Rails and Python, with Rails as the production base.
• Experience building eval pipelines, dashboards, or CI/CD for evals.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Able to run many experiments and learn from results in a dynamic environment.
• Comfortable with ambiguity and rapidly adopting new tech.
• Strong alignment with company values and fluent English (CEFR C2/ILR 5).

Responsibilities

• Build and own our evaluation infrastructure, including CI/CD pipelines, scorers, and datasets.
• Diagnose quality breakdowns across the agent pipeline and prioritize fixes.
• Grow annotation workflows and AI-generated or simulated conversations.
• Run experiments across prompts, models, and agent harness; optimize cost and latency.
• Lead and grow the AI Quality engineering team while remaining hands-on in code.
• Collaborate closely with AI Core engineering to ensure quality improvements.

Skills

LLM-powered agents

Evaluation frameworks

Experimentation

Leadership

Tools

Ruby on Rails

Python

BrainTrust

LangSmith

Job description

Circle is a fully remote, global team building AI-powered, all-in-one platforms for creators and businesses. We seek a Lead Engineer to own evaluation infrastructure, diagnose quality in AI agents, and lead an expanding AI Quality engineering team while staying hands-on in code.

The role emphasizes building eval pipelines, datasets, experiments, and collaborating with AI Core to improve reliability, speed, and cost efficiency across production systems.