Remote AI Platform Lead & Quality Engineer

Lever, Inc.; South Africa; Remote; ZAR 2,662,000 - 3,161,000; Full time

Benefits offered by this job

Annual cash compensation of $170,000

Equity and refresh grants

35 days paid time off

Fully remote work environment

Flexible work organization

Twice-yearly company retreats

Health and wellbeing benefits

Opportunity to lead AI Quality team

Exposure to advanced AI agents and AIQ

Global distributed team

Job summary

Lever, Inc. is seeking a Lead Engineer, AI Platform based in South Africa to lead the engineering foundation behind reliable, scalable AI-powered features.

You will build evaluation frameworks, observability tooling, and diagnostic infrastructure across complex agent workflows, combining hands-on software engineering with technical leadership and people management. You will design datasets, run experiments across models and prompts, and optimize AI systems for cost, latency, and user experience.

Qualifications

• 7+ years of experience building and shipping production software, including LLM-powered agents.
• Experience with complex AI systems using multiple tools, planning, or sub-agents.
• Proven ability to ship software and quantify its reliability.
• Experience with Ruby on Rails and/or Python.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Familiarity with evaluation frameworks such as Braintrust, LangSmith, or similar tools.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Ability to learn quickly and use empirical results to guide technical decisions.
• Strong leadership and people-management capabilities.
• Excellent English proficiency (CEFR C2).

Responsibilities

• Design, build, and own evaluation infrastructure, including CI/CD pipelines, scorers, datasets, and systems for assessing AI agents.
• Develop observability and diagnostic capabilities to identify quality issues across planning, execution, tool selection, and complex agent trajectories.
• Investigate failures across AI workflows and turn findings into prototypes, improvements, or priorities for AI engineering teams.
• Build and expand datasets through human annotation, AI-generated examples, and simulated conversations.
• Develop structured experimentation frameworks for prompts, models, and agent harnesses; evaluate new/open-source models against baselines.
• Identify opportunities to reduce AI system cost and latency via model selection, caching, and routing.
• Set direction and manage day-to-day priorities for the AI Quality team while remaining hands-on.
• Partner with AI Core teams to ensure measurable improvements to AI products.
• Establish engineering practices and evaluation approaches for reliable, scalable production AI systems.

Skills

LLM-powered agents

Observability infrastructure

Leadership

English proficiency

Tools

Braintrust

LangSmith

Job description

Lever, Inc. is seeking a Lead Engineer, AI Platform based in South Africa to lead the engineering foundation behind reliable, scalable AI-powered features.

You will build evaluation frameworks, observability tooling, and diagnostic infrastructure across complex agent workflows, combining hands-on software engineering with technical leadership and people management. You will design datasets, run experiments across models and prompts, and optimize AI systems for cost, latency, and user experience.