Senior Quality Engineer: AI Testing & Tooling (Remote)

Lever, Inc.; Australia; Remote; AUD 207,000 - 333,000; Full time

Benefits offered by this job

Fully remote work

Home office budget

Flexible time off

Healthcare benefits

Job summary

Partner Company in Australia seeks a Senior Software Engineer, Quality to advance AI-powered testing and intelligent automation. You will design, build, and operate AI agents for testing, while owning the infrastructure, data, and guardrails needed for reliable, scalable test execution.

The role combines hands-on software engineering with test infrastructure and distributed systems, contributing to evaluation methods for non-deterministic AI capabilities across a global, fully remote engineering

Qualifications

• 5+ years of professional software engineering experience with a track record of shipping production software.
• Hands-on experience building and owning test infrastructure, developer tooling, internal platforms, or comparable systems used by engineers.
• Practical experience with LLMs or agent frameworks, including prompt and context design, tool use, non-deterministic behavior, and evaluation of AI-generated outputs.
• Strong understanding of distributed systems and continuous integration at scale, including concurrency, state, timing, and reliability.
• Ability to influence engineering teams without direct management authority and drive adoption of tools or practices.
• Ability to understand the product deeply to build effective quality solutions.
• Strong engineering judgment and the ability to balance quality, maintainability, delivery speed, and business needs.
• Excellent collaboration and communication skills in a distributed, cross-functional environment.
• Preferred experience with LLM evaluation/benchmarking, open-source software development, or developer tooling and platform engineering.
• Experience with Java, TypeScript, or Python, as well as Kubernetes and Helm, is advantageous.

Responsibilities

• Build and operate AI agents across the testing lifecycle, including design, generation, maintenance, execution, and failure triage.
• Own the infrastructure required to make agent-driven testing dependable, including test environments, data, tool interfaces, guardrails, runtime management, and cost controls.
• Establish reliable methods for evaluating agent output quality so engineering teams can trust automated results.
• Apply AI-driven evaluation practices to testing the platform's non-deterministic and AI-powered capabilities.
• Collaborate with product engineering teams to keep testing tools aligned with real feature needs.
• Develop reusable testing patterns, tooling, and enablement practices for engineers to own quality.
• Drive adoption of quality engineering capabilities across teams.
• Contribute to technical decisions, system design, and continuous improvement of testing infrastructure.

Skills

Collaboration

Communication

Tools

Kubernetes

Helm

Java

TypeScript

Python

Job description

Partner Company in Australia seeks a Senior Software Engineer, Quality to advance AI-powered testing and intelligent automation. You will design, build, and operate AI agents for testing, while owning the infrastructure, data, and guardrails needed for reliable, scalable test execution.

The role combines hands-on software engineering with test infrastructure and distributed systems, contributing to evaluation methods for non-deterministic AI capabilities across a global, fully remote engineering