Lead AI Quality Engineer & Evaluation Platform

Circle; Pakistan; Remote; PKR 42,359,000 - 51,772,000; Full time

Job summary

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses. We’re a fully remote company with a global team and a strong focus on async collaboration.

We’re seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for Circle's AI Agents, while guiding a growing AI Quality team. You’ll work across prompts, models, and routing, shaping experiments and datasets to improve reliability, efficiency, and

Qualifications

• Over 7 years of experience shipping production software, with LLM-powered agents.
• Proficient in Ruby on Rails and Python for production systems.
• Experience building evaluation or observability infrastructure for ML/AI.
• Experience designing datasets and labeling pipelines for ML/AI evaluation.
• Able to run many experiments in a dynamic environment and learn quickly.

Responsibilities

• Build and own evaluation infrastructure, including CI/CD pipelines and datasets.
• Diagnose quality breakdowns across the agent pipeline and prioritize fixes.
• Grow annotation workflows and AI-generated/simulated conversations.
• Run structured experiments across prompts, models, and routing by plan complexity.
• Lead and grow the AI Quality engineering team while staying hands-on.
• Collaborate with AI Core engineering to ensure real quality improvements.

Skills

Production software

LLM-powered agents

Experimentation

English proficiency

Tools

Ruby on Rails

Python

LangSmith

Braintrust

CI/CD

Job description

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses. We’re a fully remote company with a global team and a strong focus on async collaboration.

We’re seeking a Lead Engineer to build evaluation frameworks, observability tooling, and diagnostic infrastructure for Circle's AI Agents, while guiding a growing AI Quality team. You’ll work across prompts, models, and routing, shaping experiments and datasets to improve reliability, efficiency, and