Senior AI Test Engineer (Sydney or Melbourne)

Automic Group

Sydney

Vor Ort

AUD 120.000 - 170.000

Vollzeit

vor 4 Stunden
Sei unter den ersten Bewerbenden

Eine vollständige Bewerbung in einer Minute

Maßgeschneiderter Lebenslauf und Anschreiben – versandbereit.

Schaffe es an den ATS-Filtern vorbei

Benefits dieser Stelle

Career growth & development
Supportive & fast paced environment
Opportunity to grow with the business

Zusammenfassung

Automic Group is seeking a Senior AI Test Engineer to lead quality assurance for AI-enabled products, agents and platforms. You will shape testing strategy, establish evaluation datasets, and ensure safe, reliable, and production-ready AI solutions in a fast-paced fintech environment.

You will work with the AI Engineering team to drive automated quality controls, contribute to release decisions, and investigate production issues while adhering to risk-based testing practices.

Qualifikationen

  • Hands-on experience designing and running tests or evaluations for generative AI, LLM, agentic or RAG applications.
  • Proven experience in quality engineering or test automation for complex production systems.
  • Experience evaluating AI response accuracy, relevance, groundedness, hallucinations and safety.
  • Experience building evaluation datasets, defining quality thresholds and automating regression tests.
  • Programming experience in Python or another language used for test automation.
  • Experience testing APIs, integrations and end-to-end workflows, including security, resilience and performance.
  • Strong analytical and communication skills; able to explain findings and make release recommendations.
  • Tertiary qualification in technology/engineering or equivalent professional experience.
  • Experience in financial services or regulated environments is desirable.

Aufgaben

  • Lead risk-based quality strategies for AI applications, agents, retrieval systems and supporting services.
  • Deliver functional and non-functional assurance across integrations and end-to-end workflows.
  • Establish AI evaluation datasets, test data, quality thresholds and automated regression coverage.
  • Assess AI response quality and safety, including hallucination, groundedness, adversarial behaviour and sensitive-data exposure.
  • Validate agent workflows, authentication, authorisation, human approvals and tenant isolation.
  • Drive automated quality controls, defect management and continuous improvement across delivery pipelines.
  • Provide evidence-based release recommendations and investigate production quality issues.

Kenntnisse

Generative AI testing
LLM testing
RAG testing
Test automation
Python
API testing
Security testing
Performance testing
Communication skills

Ausbildung

Bachelor’s degree in technology/engineering

Jobbeschreibung

Senior AI Test Engineer (Sydney or Melbourne)

Posted: 01/09/2026

Closing Date: 05/12/2026

Job Type: Permanent - Full Time

Location: Sydney

Job Category: IT

AutomicGroup is Australia’s market leader in technology-driven investor administration. Combining proprietary cloud-native technology with expert service,Automicsimplifies investor administration and governance for companies and funds. We provide the market's leading registry, employee share plans, IR analytics, and fund administration services for over 1,400 ASX-listed and unlisted organisations.

Automic’sculture is collaborative, inclusive, and purpose driven. It is grounded in our mission to connect companies and funds to their investors through technology, service, and trust, and guided by our values ofOneAutomic, Go Beyond, Empowered, and Accountable.

Job Description

About the Role

We'relooking for aSenior AI Test Engineer to provide independent assurance thatAutomic’sAI-enabled products, agents and platforms are reliable,secureand production-ready.

Working closely with our AI Engineering team,you’llshape quality strategy,establishrobust AI evaluation and testing practices, and provide evidence-based release and residual-risk advice.You’llplay a key role in ensuring our AI solutions deliver safe,reliableand high-quality outcomes.

This role requires direct experience testing AI systems. Using AI coding assistants or generative AI to write test cases, scripts or documentation alone does not meet this requirement.

Key Responsibilitiesinclude:

Lead risk-based quality strategies for AI applications, agents, retrievalsystemsand supporting services.

Deliver functional and non-functional assurance across integrations and end-to-end workflows.

Establish AI evaluation datasets, test data, qualitythresholdsand automated regression coverage.

Assess AI response quality and safety, including hallucination,groundedness, adversarial behaviour and sensitive-data exposure.

Validate agent workflows, authentication, authorisation, humanapprovalsand tenant isolation.

Drive automated quality controls, defectmanagementand continuous improvement across delivery pipelines.

Provide evidence-based release recommendations and investigate production quality issues.

Desired Skills and Experience

About you!

To be successful in this role,you will have:

Hands-on experience designing and running tests or evaluations for generative AI, LLM, agentic or RAG applications, with clear examples of your personal contribution. This is essential; using AI tools to assist conventional software testing alone is not sufficient.

Proven experience in quality engineering or test automation for complex production systems.

Practical experience evaluating AI response accuracy, relevance, groundedness, hallucinations and safety.

Experience building evaluation datasets, defining quality thresholds and automating regression tests that account for variation in AI outputs.

Programming experience in Python or another language used for test automation and service testing.

Experience testing APIs, integrations and end-to-end workflows, including security, resilience and performance.

Strong analytical and communication skills, with the ability to explain findings, work through defects with engineers and make evidence-based release recommendations.

A tertiary qualification in technology, engineering or a related discipline, or equivalent professional experience.

Experience in financial services or another regulated environment is desirable.

You’llbe part of a team whereoperational excellence, collaboration, and continuous improvementare genuinely valued. This is an opportunity to deepen yourexpertise, contribute to meaningful outcomes, and grow your career within a high-performing, purpose-led fintech environment.

Key Benefits include:

Professional career growth & development

Supportive & fast paced environment

Opportunity to grow with the business

Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Senior Ai Test Engineer (Sydney Or Melbourne)
Senior Ai Test Engineer (Sydney Or Melbourne)

Automic Group • Sydney

Vor Ort
AUD 110.000 - 160.000
Professional career growth
Supportive & fast paced environment
Opportunity to grow with the business
Senior AI Test Engineer
Senior AI Test Engineer

Clicks IT Recruitment • Brisbane

Hybrid
AUD 110.000 - 140.000
Senior AI Engineer - AI Innovation Office
Senior AI Engineer - AI Innovation Office

Westpac Group • Sydney

Hybrid
AUD 180.000 - 240.000
Senior AI Engineer
Senior AI Engineer

Peoplebank • Melbourne

Hybrid
AUD 150.000 - 230.000
Senior AI Engineer
Senior AI Engineer

Peoplebank Australia VIC • Melbourne

Hybrid
AUD 180.000 - 350.000
Hybrid work arrangement
Bonus
Senior Engineer
Senior Engineer

Westpac Group • Sydney

Hybrid
AUD 180.000 - 280.000
Hybrid working
Senior Ai Engineer - Ai Innovation Office
Senior Ai Engineer - Ai Innovation Office

Westpac Group • Melbourne

Hybrid
AUD 140.000 - 210.000
Senior Engineering Quality Assurance Analyst - Advice Technology
Senior Engineering Quality Assurance Analyst - Advice Technology

Insignia Financial Ltd • Sydney

Vor Ort
AUD 120.000 - 150.000
Senior AI Engineer - AI Innovation Office ...
Senior AI Engineer - AI Innovation Office ...

Westpac Group • Sydney

Hybrid
AUD 180.000 - 260.000
Senior AI Engineer - AI Innovation Office
Senior AI Engineer - AI Innovation Office

Westpac Banking Corporation • Sydney

Hybrid
AUD 180.000 - 240.000
Top-Standorte