Senior AI QA Engineer - Risk-Based Testing & Reliability

Automic Group; Sydney; On-site; AUD 150,000 - 190,000; Full time

Benefits offered by this job

Career growth

Supportive environment

Growth opportunities

Job summary

Automic Group is seeking a Senior Quality Assurance Engineer – AI to drive independent quality assurance for AI-enabled products. You will shape quality strategy with the AI Engineering team, establish evaluation datasets, and ensure safe, reliable delivery of AI solutions across our platforms.

You will lead risk-based testing, implement automated quality controls, and provide evidence-based release recommendations while collaborating with cross-functional teams in a dynamic fintech environment.

Qualifications

• Proven experience in quality engineering or test automation for complex production systems.
• Practical experience testing AI, ML, generative AI, agentic or retrieval-based applications.
• Strong capability in AI quality assurance, evaluation and risk-based testing.
• Programming experience in Python or another language used for test automation and service testing.
• Experience across automated service, integration, security, resilience and non-functional testing.
• Strong analytical skills and the ability to make evidence-led quality and release decisions.
• Excellent communication skills with the ability to influence and coach cross-functional delivery teams.

Responsibilities

• Lead risk-based quality strategies for AI applications, agents, retrieval systems and supporting services.
• Deliver functional and non-functional assurance across integrations and end-to-end workflows.
• Establish AI evaluation datasets, test data, quality thresholds and automated regression coverage.
• Assess AI response quality and safety, including hallucination, groundedness, adversarial behaviour and sensitive-data exposure.
• Validate agent workflows, authentication, authorisation, human approvals and tenant isolation.
• Drive automated quality controls, defect management and continuous improvement across delivery pipelines.
• Provide evidence-based release recommendations and investigate production quality issues.

Skills

Quality engineering

Test automation

AI testing

Python

Non-functional testing

Risk-based testing

Communication

Education

Tech/Engineering degree

Job description

Automic Group is seeking a Senior Quality Assurance Engineer – AI to drive independent quality assurance for AI-enabled products. You will shape quality strategy with the AI Engineering team, establish evaluation datasets, and ensure safe, reliable delivery of AI solutions across our platforms.

You will lead risk-based testing, implement automated quality controls, and provide evidence-based release recommendations while collaborating with cross-functional teams in a dynamic fintech environment.