AI Validation Scientist for Generative & Agentic Systems

Commonwealth Bank of Australia; Sydney; On-site; AUD 110,000 - 150,000; Full time

Job summary

Commonwealth Bank of Australia is seeking a Data Scientist in AI & Model Validation to independently test and challenge ML/AI use cases across the Group. You will evaluate data, methodologies and outcomes, and build reusable evaluations and test harnesses to support scalable validation.

You will collaborate with model developers, product owners and stakeholders while maintaining independence and rigorous validation practices.

Qualifications

• Advanced Python and CI/CD with GitHub; SQL/R/TypeScript exposure is valuable.
• Frameworks/tools such as LangChain, LangGraph, OpenAI Agents SDK, Pydantic.
• Knowledge of Generative AI and agentic architectures (RAG, tool use, guardrails, memory).
• Hands-on experience building or evaluating AI/GenAI solutions, including synthetic data.
• Experience using AI coding agents like Claude Code or Codex; building reusable skills, agents or plugins.
• Working knowledge of traditional ML approaches (regression, random forests, XGBoost) and evaluation methods.
• Ability to read unfamiliar code, reproduce evaluation pipelines and analyze trace data.
• Strong communication to explain complex analysis and risks to diverse audiences.

Responsibilities

• Independently deliver components of AI/ML validation, including analysis, testing and evaluation.
• Review data, methodologies and model outcomes; investigate unexpected behavior and weaknesses.
• Design evaluations for Generative AI and agentic systems (RAG, tooling, guardrails).
• Build reproducible test harnesses and automation to improve validation efficiency.
• Navigate unfamiliar codebases, reproduce pipelines and inspect trace data.
• Provide evidence-based technical challenge and convey findings to stakeholders.
• Share knowledge to advance AI validation practices and tooling.

Skills

Advanced Python

GitHub & CI/CD

SQL

R

TypeScript

LangChain

OpenAI Tools

Communication

Problem Solving

Tools

LangChain

LangGraph

OpenAI Agents SDK

Pydantic

Langfuse

HoneyHive

Job description

Commonwealth Bank of Australia is seeking a Data Scientist in AI & Model Validation to independently test and challenge ML/AI use cases across the Group. You will evaluate data, methodologies and outcomes, and build reusable evaluations and test harnesses to support scalable validation.

You will collaborate with model developers, product owners and stakeholders while maintaining independence and rigorous validation practices.