Lead AI Quality Engineer & Platform Architect

Circle

South Africa

Remote

ZAR 2,527,000 - 3,088,000

Full time

12 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Equity grants
35 days paid time off
Remote-first culture

Job summary

Circle is seeking a Lead Engineer for AI Quality Engineering to build the evaluation framework and observability tooling that ensures our AI agents operate reliably, efficiently, and at scale. You will lead a hands-on team, drive experiments, and partner with AI Core to improve production AI systems.

The role emphasizes designing evaluation datasets, workflows, and CI/CD for ML/AI, with a remote-first culture that values autonomy, documentation, and cross-time-zone collaboration.

Qualifications

  • 7+ years of experience building and shipping production software.
  • Experience with LLM-powered agents in a product setting.
  • Proficiency in Ruby on Rails and/or Python in production systems.
  • Experience building eval pipelines, observability, or CI/CD for ML/AI.
  • Experience designing datasets and labeling pipelines for ML/AI evaluation.
  • Strong English communication and collaboration in remote, fast-paced environments.

Responsibilities

  • Build and own the evaluation infrastructure: CI/CD pipelines, scorers, and datasets to validate Circle's AI agents across single calls and multi-turn conversations.
  • Diagnose where quality breaks down across the agent pipeline and translate findings into actionable prototypes or priorities.
  • Grow evaluation datasets with annotation workflows and AI-generated or simulated conversations to accelerate coverage.
  • Run structured experiments across prompts, models, and agent harness to optimize performance and cost.
  • Lead and grow the AI Quality engineering team; set technical direction while staying hands-on.
  • Collaborate with AI Core engineering to ensure changes improve quality, not just ship features.

Skills

LLM-powered agents
Experimentation in dynamic env.
English proficiency

Tools

Ruby on Rails
Python
Braintrust
LangSmith

Job description

Circle is seeking a Lead Engineer for AI Quality Engineering to build the evaluation framework and observability tooling that ensures our AI agents operate reliably, efficiently, and at scale. You will lead a hands-on team, drive experiments, and partner with AI Core to improve production AI systems.

The role emphasizes designing evaluation datasets, workflows, and CI/CD for ML/AI, with a remote-first culture that values autonomy, documentation, and cross-time-zone collaboration.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Engineer, Ai Platform
Lead Engineer, Ai Platform

Circle • South Africa

Remote
ZAR 2,527,000 - 3,088,000
Equity grants
35 days paid time off
Remote-first culture
Principal QA Platform Engineer - AI-Driven Testing Leader
Principal QA Platform Engineer - AI-Driven Testing Leader

Impact • Cape Town

On-site
ZAR 1,500,000 - 2,100,000
Flexible PTO
Mental health support
RSUs
Senior AI Engineering Architect & Platform Lead
Senior AI Engineering Architect & Platform Lead

Datafin It Recruitment • Cape Town

On-site
ZAR 1,200,000 - 2,000,000
QA Chapter Lead — Engineer-Driven Quality for IO & AI
QA Chapter Lead — Engineer-Driven Quality for IO & AI

impact.com • Cape Town

On-site
ZAR 900,000 - 1,300,000
Flexible Working
Mental health support
RSUs
+3
AI Engineer
AI Engineer

Sabenza IT & Recruitment • Pretoria

On-site
ZAR 600,000 - 800,000
QA Chapter Lead: Elevate Quality Across IO & AI
QA Chapter Lead: Elevate Quality Across IO & AI

Impact • Cape Town

On-site
ZAR 850,000 - 1,300,000
Flexible Working
Health and Wellness
RSUs
+3
Remote Lead ML Engineer: Architect & Mentor for Scalable AI
Remote Lead ML Engineer: Architect & Mentor for Scalable AI

Placements24 • Kempton Park

Remote
ZAR 1,200,000 - 1,900,000
Equity options
Health coverage
Remote stipend
+3
Global AI Quality & Observability Engineer
Global AI Quality & Observability Engineer

Plan.Net CreativeNest South Africa • Wes-Kaap

On-site
ZAR 900,000 - 1,300,000
AI Engineer
AI Engineer

Platform45 • Johannesburg

On-site
ZAR 900,000 - 1,300,000
AI Engineer
AI Engineer

Ats Client • Pretoria

On-site
ZAR 1,000,000 - 1,800,000