AI Validation Scientist: Generative & Agentic AI

Commonwealth Bank; Sydney; On-site; AUD 90,000 - 130,000; Full time

Job summary

Commonwealth Bank is seeking a data scientist focused on AI and model validation. You will independently test and challenge ML and Generative AI use cases, build evaluation harnesses, and validate architectures including RAG, tool use, and guardrails.

You’ll work with model developers and stakeholders to ensure rigorous, scalable validation practices. You will design thorough assessments, reproduce evaluation pipelines, and communicate complex findings clearly.

Qualifications

• Hands-on Python programming with data science experience.
• Experience with AI frameworks and evaluation tools.
• Ability to read unfamiliar code and reproduce evaluation pipelines.
• Experience with LangChain, LangGraph and OpenAI Agents SDK is advantageous.
• Strong communication of complex results to technical and non-technical audiences.
• Familiarity with traditional ML methods and model validation practices.

Responsibilities

• Independently deliver defined components of AI and ML validation.
• Review data, methodologies, and model outcomes to identify weaknesses.
• Design and execute evaluations for Generative AI and agentic systems.
• Build reproducible test harnesses and automation.
• Navigate unfamiliar codebases and reproduce evaluation pipelines.
• Provide evidence-based technical challenges and conclusions.
• Communicate findings clearly to stakeholders.
• Share knowledge to advance AI validation practices.

Skills

Advanced Python

GitHub & CI/CD

SQL/R exposure

LangChain

LangGraph

OpenAI Agents SDK

Pydantic

Langfuse

HoneyHive

Agentic AI basics

Tools

GitHub

CI/CD tooling

LangChain

LangGraph

OpenAI Agents SDK

Pydantic

Langfuse

HoneyHive

Job description

Commonwealth Bank is seeking a data scientist focused on AI and model validation. You will independently test and challenge ML and Generative AI use cases, build evaluation harnesses, and validate architectures including RAG, tool use, and guardrails.

You’ll work with model developers and stakeholders to ensure rigorous, scalable validation practices. You will design thorough assessments, reproduce evaluation pipelines, and communicate complex findings clearly.