Principal Data Engineer - Lead AI-Driven Data Platform

Medibank; City of Melbourne; On-site; AUD 180,000 - 260,000; Full time

Job summary

Medibank is seeking a Principal Data Engineer to define and lead the technical strategy for its enterprise data platforms. You will shape Databricks-based lakehouse architectures, collaborate with architects, data scientists, and business leaders, and drive modern DataOps practices at scale.

You will mentor engineering teams, establish standards, and accelerate delivery across globally distributed squads while championing data governance and security across the platform.

Qualifications

• 10+ years in Data Engineering, Software Engineering, or a related field.
• 5+ years working with Databricks or large-scale Spark platforms.
• 5+ years delivering cloud-native data solutions.
• Proven ability to define technical strategy and influence enterprise architecture decisions.

Responsibilities

• Lead design and delivery of scalable lakehouse architectures and data platforms using Databricks.
• Develop reusable frameworks, accelerators, and reference implementations.
• Partner with Engineers and Architects to solve complex business challenges with data-driven solutions.
• Mentor engineers and promote engineering excellence across teams.
• Communicate complex concepts to both technical and non-technical audiences.

Skills

Databricks

Unity Catalog

MLflow

Databricks SQL

PySpark

Spark SQL

Data modelling

Data Vault

Feature store design

Airflow

Databricks Workflows

Azure Data Factory

MLOps

CI/CD for ML

GitHub Actions

Azure DevOps

Jenkins

Terraform

ARM/Bicep

CloudFormation

Data governance

Metadata management

Python

SQL

REST APIs

Kafka

Kinesis

Microservices

Tools

Databricks

Airflow

Azure Data Factory

GitHub Actions

Azure DevOps

Jenkins

Terraform

Job description

Medibank is seeking a Principal Data Engineer to define and lead the technical strategy for its enterprise data platforms. You will shape Databricks-based lakehouse architectures, collaborate with architects, data scientists, and business leaders, and drive modern DataOps practices at scale.

You will mentor engineering teams, establish standards, and accelerate delivery across globally distributed squads while championing data governance and security across the platform.