AI Evaluation & Governance Lead

University of Sydney; Sydney; Vor Ort; AUD 137.000 - 167.000; Vollzeit

Zusammenfassung

University of Sydney seeks a Manager, AI Evaluation to lead the assessment, testing and governance of generative AI solutions. You will establish an evaluation framework across the AI Hub, drive pre-release testing, governance approvals and continuous monitoring of AI services in production.

You will work with product, delivery and technical teams, plus ICT, Legal and Privacy to embed best-practice evaluation processes, identify risks and ensure high-quality outcomes across the University.

Qualifikationen

• Postgraduate qualifications in a relevant field or equivalent experience.
• “Demonstrated experience in data science, machine learning, NLP and AI solutions in complex organisations.”
• “Strong knowledge of generative AI, LLMs, agentic AI frameworks and modern data science methodologies.”
• “Analytical, problem-solving and data analysis skills with ability to interpret complex data.”
• “Experience developing evaluation, monitoring and training frameworks with attention to detail.”
• “Excellent communication, planning and stakeholder engagement skills in Agile environments.”

Aufgaben

• Lead development and continuous improvement of AI evaluation frameworks, methodologies and standards.
• Govern evaluation datasets, metrics and testing approaches for performance, reliability, safety and quality.
• Oversee pre-release evaluation, testing and assurance activities with independent recommendations for governance approvals.
• Monitor production AI services, perform regression testing and assess drift and opportunities for improvement.
• Collaborate with technical teams, data stewards and governance bodies to ensure data integrity and compliance.
• Provide leadership and capability development in AI evaluation and communicate findings across the AI Hub.

Kenntnisse

AI evaluation

Data science

Machine learning

Natural language processing

LLMs

Governance

Stakeholder engagement

Agile

Ausbildung

Postgraduate qualifications in a relevant field

Equivalent education and experience

Tools

AI evaluation frameworks

Data analysis methods

Jobbeschreibung

University of Sydney seeks a Manager, AI Evaluation to lead the assessment, testing and governance of generative AI solutions. You will establish an evaluation framework across the AI Hub, drive pre-release testing, governance approvals and continuous monitoring of AI services in production.

You will work with product, delivery and technical teams, plus ICT, Legal and Privacy to embed best-practice evaluation processes, identify risks and ensure high-quality outcomes across the University.