Principal Data Engineer: Databricks & AI Platform Leader

Medibank Private Limited; Australia; On-site; AUD 180,000 - 240,000; Full time

Job summary

Medibank Private Limited is seeking a Principal Data Engineer to lead the strategic direction and engineering excellence of our enterprise data platforms. You will drive cloud-native data pipelines, Databricks platforms, and AI/ML enablement while mentoring engineering teams across the organisation.

The role focuses on scalable lakehouse architectures, data governance, and driving modern platform practices to power analytics and customer experiences enterprise-wide.

Qualifications

• Extensive Databricks experience including Unity Catalog and MLflow.
• Strong PySpark, Spark SQL and distributed processing skills.
• Deep data modelling knowledge (dimensional/ Data Vault, feature store).
• Experience with DataOps, MLOps, CI/CD for ML and data platforms.

Responsibilities

• Define and lead technical roadmap for Databricks platform and data pipelines.
• Drive architecture and implementation of lakehouse solutions at scale.
• Mentor engineers and promote best practices across data teams.
• Collaborate with data scientists and business stakeholders to enable analytics and ML.
• Champion DataOps/MLOps, observability, testing, and automation.

Skills

Databricks

PySpark

Data modelling

MLOps

Cloud data platforms

Python

SQL

REST APIs

Airflow

CI/CD tooling

Education

10+ years Data/Software Eng

Tools

Databricks Unity Catalog

MLflow

Terraform

Azure Data Factory

Job description

Medibank Private Limited is seeking a Principal Data Engineer to lead the strategic direction and engineering excellence of our enterprise data platforms. You will drive cloud-native data pipelines, Databricks platforms, and AI/ML enablement while mentoring engineering teams across the organisation.

The role focuses on scalable lakehouse architectures, data governance, and driving modern platform practices to power analytics and customer experiences enterprise-wide.