Data Platform Lead — Databricks, AI/ML & DataOps

Medibank; Victoria; On-site; AUD 180,000 - 260,000; Full time

Job summary

Medibank seeks a Principal Data Engineer to lead the technical strategy for enterprise data platforms and Databricks-based lakehouse architectures. You will mentor engineering teams, set standards, and drive AI/ML enablement across the organisation.

You will partner with architects, data scientists and business leaders to deliver scalable, secure data solutions powering analytics and customer experiences, while advancing DataOps and MLOps practices across global teams.

Qualifications

• Expert-level experience with Databricks including Unity Catalog, MLflow, and Databricks SQL.
• Advanced PySpark, Spark SQL, and distributed data processing expertise.
• Strong data modelling methods: dimensional, Data Vault, feature store design.
• Experience with orchestration tools like Airflow, Databricks Workflows, or Azure Data Factory.
• Proven ability to productionise ML workloads at enterprise scale and CI/CD for ML.

Responsibilities

• Define and drive the technical roadmap for Databricks platform and AI/ML capabilities.
• Lead design and delivery of scalable lakehouse architectures and data platforms.
• Mentor engineers and promote DataOps and MLOps practices.
• Partner with architects, data scientists, and business leads to solve complex challenges.
• Champion engineering standards, testing, observability, and security controls.

Skills

Databricks

Unity Catalog

MLflow

Databricks SQL

PySpark

Spark SQL

Data modelling

Data Vault

Feature store design

Airflow

Azure Data Factory

GitHub Actions

Azure DevOps

Jenkins

Terraform

CloudFormation

REST APIs

Kafka

Kinesis

Tools

Databricks

Azure

Hadoop/Spark

Job description

Medibank seeks a Principal Data Engineer to lead the technical strategy for enterprise data platforms and Databricks-based lakehouse architectures. You will mentor engineering teams, set standards, and drive AI/ML enablement across the organisation.

You will partner with architects, data scientists and business leaders to deliver scalable, secure data solutions powering analytics and customer experiences, while advancing DataOps and MLOps practices across global teams.