AI Validation Scientist: Generative & Agentic AI

CommBank; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

CommBank is seeking a Data Scientist to join AI & General Model Validation (AIGMV). You’ll independently evaluate ML and Generative AI use cases, focusing on data, methods, assumptions and outcomes to support safe, scalable AI adoption.

You’ll collaborate with model developers, product owners and business stakeholders to design evaluations, build reusable test harnesses, and clearly communicate risks and findings to diverse audiences.

Qualifications

• Curious, hands‑on Data Scientist who enjoys probing how AI systems behave.
• Strong analytical judgment and attention to detail; able to investigate unexpected results.
• Experience with evaluating data, methodologies, and model outcomes; ability to communicate risks clearly.

Responsibilities

• Independently deliver defined components of AI/ML validation, including analysis, testing and evaluation.
• Review data, methodologies, assumptions and outcomes; investigate unexpected behavior and weaknesses.
• Design and execute evaluations for Generative AI and agentic systems (RAG, tools, guardrails, routing, memory).
• Build reproducible test harnesses, reusable evaluations and automation to improve validation efficiency.
• Navigate unfamiliar codebases, reproduce evaluation pipelines and inspect trace data to understand model behavior.
• Provide evidence‑based technical challenge and present findings to stakeholders.
• Share knowledge and contribute to AI validation practices, tooling and standards.

Skills

Python

GitHub/CI‑CD

SQL/R/TS

LangChain/ LangGraph

OpenAI Agents SDK

Generative AI

Communication

Education

Bachelor’s degree in CS or related

Tools

LangChain

LangGraph

Pydantic

Langfuse

HoneyHive

Job description

CommBank is seeking a Data Scientist to join AI & General Model Validation (AIGMV). You’ll independently evaluate ML and Generative AI use cases, focusing on data, methods, assumptions and outcomes to support safe, scalable AI adoption.

You’ll collaborate with model developers, product owners and business stakeholders to design evaluations, build reusable test harnesses, and clearly communicate risks and findings to diverse audiences.