Manager, AI Evaluation

University of Sydney; Sydney; On-site; AUD 137,000 - 167,000; Full time

Job summary

University of Sydney seeks a Manager, AI Evaluation to lead the assessment, testing and governance of generative AI solutions. You will establish an evaluation framework across the AI Hub, drive pre-release testing, governance approvals and continuous monitoring of AI services in production.

You will work with product, delivery and technical teams, plus ICT, Legal and Privacy to embed best-practice evaluation processes, identify risks and ensure high-quality outcomes across the University.

Qualifications

• Postgraduate qualifications in a relevant field or equivalent experience.
• “Demonstrated experience in data science, machine learning, NLP and AI solutions in complex organisations.”
• “Strong knowledge of generative AI, LLMs, agentic AI frameworks and modern data science methodologies.”
• “Analytical, problem-solving and data analysis skills with ability to interpret complex data.”
• “Experience developing evaluation, monitoring and training frameworks with attention to detail.”
• “Excellent communication, planning and stakeholder engagement skills in Agile environments.”

Responsibilities

• Lead development and continuous improvement of AI evaluation frameworks, methodologies and standards.
• Govern evaluation datasets, metrics and testing approaches for performance, reliability, safety and quality.
• Oversee pre-release evaluation, testing and assurance activities with independent recommendations for governance approvals.
• Monitor production AI services, perform regression testing and assess drift and opportunities for improvement.
• Collaborate with technical teams, data stewards and governance bodies to ensure data integrity and compliance.
• Provide leadership and capability development in AI evaluation and communicate findings across the AI Hub.

Skills

AI evaluation

Data science

Machine learning

Natural language processing

LLMs

Governance

Stakeholder engagement

Agile

Education

Postgraduate qualifications in a relevant field

Equivalent education and experience

Tools

AI evaluation frameworks

Data analysis methods

Job description

Full-time continuing opportunity drive the evaluation and quality assurance of generative AI solutions, ensuring robust governance, performance and continuous improvement across the AI Hub

Base Salary $152,045+ 17% superannuation

About the opportunity

As the Manager, AI Evaluation, you will lead the assessment, testing and governance of generative AI solutions, ensuring they are safe, effective and fit for deployment and ongoing use. You will establish and oversee a consistent evaluation framework across the AI Hub, driving pre-release testing, governance approvals, performance measurement and continuous monitoring of AI services in production. Working closely with product, delivery and technical teams, as well as key stakeholders across ICT, Legal, Privacy and data stewardship functions, you will play a critical role in embedding best-practice evaluation processes, identifying risks, and ensuring AI solutions continue to deliver reliable and high-quality outcomes across the University.

Key responsibilities

• lead the development and continuous improvement of AI evaluation frameworks, methodologies and standards to ensure consistent, rigorous assessment across the AI Hub portfolio design
• curate and govern evaluation datasets, metrics and testing approaches that measure performance, reliability, safety and quality of AI solutions
• oversee pre-release evaluation, testing and assurance activities, providing independent recommendations and evidence to support governance approvals and deployment decisions
• manage ongoing monitoring, regression testing and performance evaluation of production AI services, identifying risks, drift and opportunities for improvement
• partner with technical teams, data stewards, governance bodies and key stakeholders to ensure effective evaluation practices, data integrity and compliance requirements are met
• provide leadership, expert advice and capability development in AI evaluation, communicating findings and driving continuous improvement across the AI Hub and broader university community

About you

• postgraduate qualifications in a relevant field, or an equivalent combination of education and experience
• demonstrated experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments
• strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies
• highly developed analytical, problem‑solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence‑based decision‑making
• experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement
• excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross‑functional teams within large or complex organisations

Sponsorship / work rights for Australia

You must have unrestricted work rights in Australia for the duration of this employment to be eligible to apply. Visa sponsorship is not available for this appointment.

Pre‑employment checks and declarations

Your employment is conditional upon the successful completion of all pre‑employment or background checks required for the role in terms satisfactory to the University. Also, to meet the University’s obligations under the National Higher Education Code to Prevent and Eliminate Gender‑Based Violence you will be asked to declare if you have been investigated for, or found to engaged in, sexual harm or gender‑based violence in the course of previous employment or in a legal process. Similarly, your ongoing employment is conditional upon the satisfactory maintenance of all relevant clearances and background check requirements. If you do not meet these conditions, the University may take any necessary step, including the termination of your employment.

EEO statement

At the University of Sydney, our shared values are trust, accountability and excellence and we strive to be a place where everyone can thrive. We are committed to creating a University community that thrives through diversity and reflects the wider community that we serve. We deliver on this through our commitment to diversity and inclusion, evidenced by our people and culture programs, as well as key strategies to increase participation and support the careers of Aboriginal and Torres Strait Islander People, women, people living with a disability, people from culturally and linguistically diverse backgrounds, and those who identify as LGBTQIA+. We welcome applications from candidates from all backgrounds. We are proud to be recognised as an Australian Workplace Equality Index (AWEI) Platinum Employer. Find out more about our work on diversity and inclusion. Applications from people of culturally and linguistically diverse backgrounds; equity target groups including women, people with disabilities, people who identify as LGBTIQ; and people of Aboriginal and Torres Strait Islander descent, are encouraged.

Applications Close

Friday 09 October 2026 11:59 PM

We reserve the right not to proceed with any appointment.