Remote Lead AI Platform Quality Engineer

Lever, Inc.; Australia; Remote; AUD 215,000 - 273,000; Full time

Benefits offered by this job

Equity in company

35 days paid time off

Fully remote work environment

Global retreats

Health and wellbeing benefits

Job summary

Lever, Inc. is seeking a senior AI Quality engineer to lead the engineering foundation for reliable, scalable AI-powered features.

You will design evaluation frameworks, observability tooling, and diagnostic systems to measure AI agents in real production environments. You will balance hands-on software development with technical leadership, guide datasets and experiments across models and prompts, and partner with AI Core teams to improve quality while optimizing cost and latency.

Qualifications

• 7+ years of experience shipping production software, including AI/LLM-powered agents.
• Experience building evaluation, observability, and benchmarking infrastructure for ML/AI systems.
• Proficiency in Ruby on Rails and/or Python; ability to quickly adopt new tech.

Responsibilities

• Design, build, and own evaluation infrastructure, including datasets, scorers, and CI/CD pipelines.
• Develop observability and diagnostic capabilities across planning, execution, and tool selection in AI agent workflows.
• Investigate failures in complex AI workflows and translate findings into concrete improvements.
• Build datasets via human annotation and simulated conversations to expand evaluation coverage.
• Define experiments for prompts, models, and agent harnesses with production baselines.

Skills

Leadership

LLM workflows

Python

Ruby on Rails

Observability

CI/CD pipelines

LangSmith

Braintrust

English proficiency

Tools

Braintrust

LangSmith

CI/CD tools

Job description

Lever, Inc. is seeking a senior AI Quality engineer to lead the engineering foundation for reliable, scalable AI-powered features.

You will design evaluation frameworks, observability tooling, and diagnostic systems to measure AI agents in real production environments. You will balance hands-on software development with technical leadership, guide datasets and experiments across models and prompts, and partner with AI Core teams to improve quality while optimizing cost and latency.