Senior AI QA Engineer - Build Safe, Reliable AI

Automic Group; Sydney; On-site; AUD 110,000 - 160,000; Full time

Benefits offered by this job

Professional career growth

Supportive & fast paced environment

Opportunity to grow with the business

Job summary

Automic Group is seeking a Senior AI Test Engineer to lead the quality strategy for AI-enabled products, agents and platforms. You will shape testing practices, establish robust evaluation frameworks and provide evidence-based release guidance to ensure safe and reliable AI solutions.

Working with our AI Engineering team, you’ll drive automated quality controls, datasets, and regression coverage, while assessing AI safety and performance across complex end-to-end workflows.

Qualifications

• Hands-on experience designing and running tests or evaluations for generative AI, LLM, agentic or RAG applications.
• Proven experience in quality engineering or test automation for complex production systems.
• Experience building evaluation datasets, defining quality thresholds and automating regression tests that account for variation in AI outputs.

Responsibilities

• Lead risk-based quality strategies for AI applications, agents, retrieval systems and supporting services.
• Deliver functional and non-functional assurance across integrations and end-to-end workflows.
• Establish AI evaluation datasets, test data, quality thresholds and automated regression coverage.
• Assess AI response quality and safety, including hallucination, groundedness, adversarial behaviour and sensitive-data exposure.
• Validate agent workflows, authentication, authorisation, human approvals and tenant isolation.
• Drive automated quality controls, defect management and continuous improvement across delivery pipelines.
• Provide evidence-based release recommendations and investigate production quality issues.

Skills

AI test design

Python

API testing

QA for AI systems

Education

Bachelor's degree in technology

Job description

Automic Group is seeking a Senior AI Test Engineer to lead the quality strategy for AI-enabled products, agents and platforms. You will shape testing practices, establish robust evaluation frameworks and provide evidence-based release guidance to ensure safe and reliable AI solutions.

Working with our AI Engineering team, you’ll drive automated quality controls, datasets, and regression coverage, while assessing AI safety and performance across complex end-to-end workflows.