Manager, Ai Evaluation

The University Of Sydney; Sydney; On-site; AUD 137,000 - 167,000; Full time

Job summary

The University of Sydney – Forest Lodge NSW is seeking a Manager, AI Evaluation to lead the assessment, testing and governance of generative AI solutions, ensuring they are safe, effective and fit for deployment. You will establish an evaluation framework across the AI Hub and collaborate with ICT, Legal and data stewardship teams.

This role drives pre-release testing, performance measurement and continuous monitoring of AI services in production, providing independent recommendations and

Qualifications

• Postgraduate qualifications in a relevant field, or an equivalent combination of education and experience.
• Experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments.
• Strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies.
• Highly developed analytical, problem-solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence-based decision-making.
• Experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement.
• Excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross-functional teams within large or complex organisations.

Responsibilities

• lead the development and continuous improvement of AI evaluation frameworks, methodologies and standards to ensure consistent, rigorous assessment across the AI Hub portfolio
• design, curate and govern evaluation datasets, metrics and testing approaches that measure performance, reliability, safety and quality of AI solutions
• oversee pre-release evaluation, testing and assurance activities, providing independent recommendations and evidence to support governance approvals and deployment decisions
• manage ongoing monitoring, regression testing and performance evaluation of production AI services, identifying risks, drift and opportunities for improvement
• partner with technical teams, data stewards, governance bodies and key stakeholders to ensure effective evaluation practices, data integrity and compliance requirements are met
• provide leadership, expert advice and capability development in AI evaluation, communicating findings and driving continuous improvement across the AI Hub and broader university community

Skills

Data science

Machine learning

Natural language processing

Generative AI

LLMs

Agentic AI frameworks

Analytical skills

Communication

Agile & stakeholder engagement

Education

Postgraduate qualifications

Job description

Manager, AI Evaluation The University of Sydney•Sydney, NSW, AU

Job Description

The University of Sydney – Forest Lodge NSW

• drive the evaluation and quality assurance of generative AI solutions, ensuring robust governance, performance and continuous improvement across the AI Hub


• Base Salary $152,045+ 17% superannuation


• Full-time continuing opportunity


• drive the evaluation and quality assurance of generative AI solutions, ensuring robust governance, performance and continuous improvement across the AI Hub


• Base Salary $152,045+ 17% superannuation

About the opportunity

As the Manager, AI Evaluation, you will lead the assessment, testing and governance of generative AI solutions, ensuring they are safe, effective and fit for deployment and ongoing use. You will establish and oversee a consistent evaluation framework across the AI Hub, driving pre-release testing, governance approvals, performance measurement and continuous monitoring of AI services in production. Working closely with product, delivery and technical teams, as well as key stakeholders across ICT, Legal, Privacy and data stewardship functions, you will play a critical role in embedding best-practice evaluation processes, identifying risks, and ensuring AI solutions continue to deliver reliable and high-quality outcomes across the University.

Your key responsibilities will be to:

• lead the development and continuous improvement of AI evaluation frameworks, methodologies and standards to ensure consistent, rigorous assessment across the AI Hub portfolio


• design, curate and govern evaluation datasets, metrics and testing approaches that measure performance, reliability, safety and quality of AI solutions


• oversee pre-release evaluation, testing and assurance activities, providing independent recommendations and evidence to support governance approvals and deployment decisions


• manage ongoing monitoring, regression testing and performance evaluation of production AI services, identifying risks, drift and opportunities for improvement


• partner with technical teams, data stewards, governance bodies and key stakeholders to ensure effective evaluation practices, data integrity and compliance requirements are met


• provide leadership, expert advice and capability development in AI evaluation, communicating findings and driving continuous improvement across the AI Hub and broader university community

About you

• postgraduate qualifications in a relevant field, or an equivalent combination of education and experience


• demonstrated experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments


• strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies


• highly developed analytical, problem-solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence-based decision-making


• experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement


• excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross-functional teams within large or complex organisations

Sponsorship / work rights for Australia

You must have unrestricted work rights in Australia for the duration of this employment to be eligible to apply. Visa sponsorship is not available for this appointment.

Pre-employment checks and declarations

Your employment is conditional upon the successful completion of all pre-employment or background checks required for the role in terms satisfactory to the University. Also, to meet the University's obligations under the National Higher Education Code to Prevent and Eliminate Gender-Based Violence you will be asked to declare if you have been investigated for, or found to engaged in, sexual harm or gender-based violence in the course of previous employment or in a legal process. Similarly, your ongoing employment is conditional upon the satisfactory maintenance of all relevant clearances and background check requirements. If you do not meet these conditions, the University may take any necessary step, including the termination of your employment.

EEO statement

At the University of Sydney, our shared values are trust, accountability and excellence and we strive to be a place where everyone can thrive. We are committed to creating a University community that thrives through diversity and reflects the wider community that we serve. We deliver on this through our commitment to diversity and inclusion, evidenced by our people and culture programs, as well as key strategies to increase participation and support the careers of Aboriginal and Torres Strait Islander People, women, people living with a disability, people from culturally and linguistically diverse backgrounds, and those who identify as LGBTQIA+.

We welcome applications from candidates from all backgrounds.

We are proud to be recognised as an Australian Workplace Equality Index (AWEI) Platinum Employer. Find out more about our work on diversity and inclusion.

The University of Sydney

The University reserves the right not to proceed with any appointment.

Applications Close

Sunday 30 August 2026 11:59 PM