Data Scientist

Commonwealth Bank of Australia; Sydney; On-site; AUD 110,000 - 150,000; Full time

Job summary

Commonwealth Bank of Australia is seeking a Data Scientist in AI & Model Validation to independently test and challenge ML/AI use cases across the Group. You will evaluate data, methodologies and outcomes, and build reusable evaluations and test harnesses to support scalable validation.

You will collaborate with model developers, product owners and stakeholders while maintaining independence and rigorous validation practices.

Qualifications

• Advanced Python and CI/CD with GitHub; SQL/R/TypeScript exposure is valuable.
• Frameworks/tools such as LangChain, LangGraph, OpenAI Agents SDK, Pydantic.
• Knowledge of Generative AI and agentic architectures (RAG, tool use, guardrails, memory).
• Hands-on experience building or evaluating AI/GenAI solutions, including synthetic data.
• Experience using AI coding agents like Claude Code or Codex; building reusable skills, agents or plugins.
• Working knowledge of traditional ML approaches (regression, random forests, XGBoost) and evaluation methods.
• Ability to read unfamiliar code, reproduce evaluation pipelines and analyze trace data.
• Strong communication to explain complex analysis and risks to diverse audiences.

Responsibilities

• Independently deliver components of AI/ML validation, including analysis, testing and evaluation.
• Review data, methodologies and model outcomes; investigate unexpected behavior and weaknesses.
• Design evaluations for Generative AI and agentic systems (RAG, tooling, guardrails).
• Build reproducible test harnesses and automation to improve validation efficiency.
• Navigate unfamiliar codebases, reproduce pipelines and inspect trace data.
• Provide evidence-based technical challenge and convey findings to stakeholders.
• Share knowledge to advance AI validation practices and tooling.

Skills

Advanced Python

GitHub & CI/CD

SQL

R

TypeScript

LangChain

OpenAI Tools

Communication

Problem Solving

Tools

LangChain

LangGraph

OpenAI Agents SDK

Pydantic

Langfuse

HoneyHive

Job description

Data Scientist | AI & Model Validation

Independently test and challenge some of the Group's most important machine learning, Generative AI and agentic AI use cases.

Build hands‑on expertise across emerging AI architectures, evaluation methods and responsible AI.

Create reusable evaluations, test harnesses and automation to help strengthen how AI is validated at scale.

Do Work That Matters

AI & General Model Validation (AIGMV) provides independent technical challenge and review of important machine learning and AI models across the Group.

As a Data Scientist in the team, you'll help assess whether AI solutions perform as intended by independently evaluating their data, methodology, assumptions and outcomes. You'll combine hands‑on technical investigation with clear, evidence‑based challenge, helping support safe and sustainable AI adoption.

See yourself in our team

You'll work closely with model developers, product owners and business stakeholders while maintaining the independence expected of a validation function.

In this role, you will:

• Independently deliver defined components of AI and ML validation, including technical analysis, testing and evaluation.
• Review data, methodologies, assumptions and model outcomes, investigating unexpected behaviour and potential weaknesses.
• Design and execute evaluations for Generative AI and agentic systems, including architectures involving RAG, tool use, guardrails, routing, memory and LLM-as-a-Judge approaches.
• Build reproducible test harnesses, reusable evaluations and automation that improve the efficiency and consistency of validation.
• Navigate unfamiliar codebases, reproduce evaluation pipelines and inspect trace data to understand model and agent behaviour.
• Provide constructive, evidence‑based technical challenge and bring your findings into broader validation conclusions.
• Communicate complex technical findings clearly to both technical and non‑technical stakeholders.
• Share knowledge and contribute to the continuing development of AI validation practices, tooling and standards.

We're interested in hearing from people who:

This role will suit a curious, hands‑on Data Scientist who enjoys getting beneath the surface of how AI systems behave. You'll bring sound analytical judgement, attention to detail and the confidence to investigate unexpected results, while staying curious as Generative AI and agentic techniques continue to evolve.

Experience across several of the following will help you succeed:

• Advanced Python skills and experience with GitHub, version control and CI/CD; SQL, R or TypeScript exposure is also valuable.
• Experience with frameworks and tools such as LangChain, LangGraph, OpenAI Agents SDK, Pydantic and evaluation or observability platforms such as Langfuse or HoneyHive.
• Practical knowledge of Generative AI and agentic architectures, including orchestrators, sub-agents, RAG pipelines, tool use, guardrails, multi‑agent routing and memory.
• Hands‑on experience building or evaluating AI and Generative AI solutions, including synthetic data; transformer fine‑tuning experience would be beneficial.
• Experience using AI coding agents such as Claude Code or Codex, including building reusable skills, agents, hooks or plugins to automate technical workflows.
• Working knowledge of traditional machine‑learning approaches such as regression, random forests and XGBoost, together with appropriate evaluation methods.
• Confidence reading unfamiliar code, reproducing evaluation pipelines, writing test harnesses and analysing trace data.
• Strong communication skills and the ability to explain complex analysis and technical risks clearly to different audiences.

If AI safety matters to you, you enjoy learning independently and you bring strength across some rather than all of these areas, we'd like to hear from you.

We're aware of some accessibility issues on this site, particularly for screen reader users. We want to make finding your dream job as easy as possible, so if you require additional support please contact HR Direct on 1800 989 696.

Advertising End Date: 10/11/2026