Lead AI Quality Engineer — Observability & Evaluation

Siemens; Lisboa; Remote; EUR 134,000 - 187,000; Full time

Benefits offered by this job

U.S.-Benchmark compensation

Equity grants

35 days paid time off

Job summary

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses, with a fully remote team and a strong focus on high-impact AI quality engineering.

We are seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for AI agents, while growing and leading the AI Quality engineering team. You will work hands-on and collaborate with AI Core engineering to deliver scalable, cost-efficient AI solutions.

Qualifications

• 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
• Comfortable in Ruby on Rails / Python or ready to pick them up quickly.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.

Responsibilities

• Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents are actually working.
• Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, and routing.
• Grow the datasets that make evaluation possible. Stand up annotation workflows and AI-generated or simulated conversations.
• Run structured experiments across prompts, models, and the agent harness to compare production baselines and drive improvements.
• Lead and grow the AI Quality engineering team with hands-on coding and clear technical direction.
• Partner closely with AI Core engineering to ensure changes improve quality, not just ship features.

Skills

Ruby on Rails

Python

LLM-powered agents

Observability tooling

Eval pipelines

CI/CD for evals

Experimentation

Team leadership

Tools

LangSmith

Braintrust

Job description

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses, with a fully remote team and a strong focus on high-impact AI quality engineering.

We are seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for AI agents, while growing and leading the AI Quality engineering team. You will work hands-on and collaborate with AI Core engineering to deliver scalable, cost-efficient AI solutions.