Principal Data Engineer

Medibank Private Limited; Australia; On-site; AUD 180,000 - 240,000; Full time

Job summary

Medibank Private Limited is seeking a Principal Data Engineer to lead the strategic direction and engineering excellence of our enterprise data platforms. You will drive cloud-native data pipelines, Databricks platforms, and AI/ML enablement while mentoring engineering teams across the organisation.

The role focuses on scalable lakehouse architectures, data governance, and driving modern platform practices to power analytics and customer experiences enterprise-wide.

Qualifications

• Extensive Databricks experience including Unity Catalog and MLflow.
• Strong PySpark, Spark SQL and distributed processing skills.
• Deep data modelling knowledge (dimensional/ Data Vault, feature store).
• Experience with DataOps, MLOps, CI/CD for ML and data platforms.

Responsibilities

• Define and lead technical roadmap for Databricks platform and data pipelines.
• Drive architecture and implementation of lakehouse solutions at scale.
• Mentor engineers and promote best practices across data teams.
• Collaborate with data scientists and business stakeholders to enable analytics and ML.
• Champion DataOps/MLOps, observability, testing, and automation.

Skills

Databricks

PySpark

Data modelling

MLOps

Cloud data platforms

Python

SQL

REST APIs

Airflow

CI/CD tooling

Education

10+ years Data/Software Eng

Tools

Databricks Unity Catalog

MLflow

Terraform

Azure Data Factory

Job description

Select how often (in days) to receive an alert:

It’s a strange thing to say, because us humans are capable of incredible things. And at Medibank, we know our greatest potential lies in the people who work with us.

We strive to make real, fundamental change, driven by a simple purpose: to create the best health and wellbeing for all of Australia.

Shape the Future of Data at Medibank

At Medibank, we're passionate about creating the best health and wellbeing experiences for our customers and communities. As we continue to evolve our enterprise data and AI capabilities, we're looking for an exceptional Principal Data Engineer to lead the technical vision and engineering excellence of our next-generation data platforms.

This is a rare opportunity for a highly experienced data engineering leader. You will drive innovation across cloud-native data platforms, Databricks, AI/ML enablement, and DataOps practices while mentoring and uplifting engineering teams across the organisation.

The Opportunity

As our Principal Data Engineer, you will define and lead the technical strategy, architecture, standards, and delivery of Medibank's enterprise data platforms.

Operating at the intersection of strategy and hands‑on engineering, you'll partner with architects, engineers, data scientists, and business leaders to design robust, scalable, and secure data solutions that power analytics, machine learning, and customer experiences across the enterprise.

You'll be a thought leader, mentor, innovator, and trusted advisor, helping engineering teams adopt modern platform practices and engineering excellence.

What You'll Be Doing

Technology Leadership

• Define and drive the technical roadmap for Medibank's Databricks platform, enterprise data pipelines, and AI/ML capabilities.
• Provide strategic technical leadership across data engineering, analytics, DataOps, and AI/ML platform initiatives.
• Evaluate emerging technologies and identify opportunities to innovate across data, cloud, and AI ecosystems.
• Act as a senior technical advisor to engineering leaders, architects, and business stakeholders.
• Establish engineering standards, frameworks, and best practices that scale across the organisation.
• Mentor and coach engineers, fostering a culture of continuous learning, collaboration, and technical excellence.

Solution Design & Delivery

• Lead the design and implementation of scalable lakehouse architectures and enterprise-grade data platforms using Databricks.
• Develop reusable frameworks, accelerators, libraries, and reference implementations to improve delivery velocity.
• Partner with Engineering Leads and Data Architects to solve complex business challenges through data-driven solutions.
• Guide teams on:
• Data modelling and architecture patterns
• Data quality frameworks
• Metadata management
• Facilitate technical workshops, platform onboarding sessions, and capability uplift across multiple delivery squads.
• Communicate complex technical concepts effectively to both technical and non-technical audiences.

Platform Excellence & Support

• Contribute to and help lead a Data Platform Centre of Excellence (CoE).
• Provide hands‑on technical expertise across multiple programs and initiatives.
• Support and coordinate engineering delivery across globally distributed and offshore teams.
• Promote consistency, reliability, maintainability, and scalability across all platform capabilities.
• Champion modern DataOps and MLOps practices, including:
• CI/CD for data and machine learning
• Automated testing
• Platform observability and monitoring
• Drive continuous improvement initiatives to reduce technical debt and improve engineering productivity.
• Develop tools and automation capabilities that streamline the development and deployment of data products.
• Influence the adoption of modern cloud-native and AI-enabled engineering practices.

About You

Essential Experience & Skills

• Expert-level experience with Databricks, including:
• Unity Catalog
• MLflow
• Databricks SQL
• Advanced expertise in PySpark, Spark SQL, and distributed data processing.
• Strong experience designing and implementing modern lakehouse architectures.
• Deep knowledge of data modelling methodologies, including dimensional modelling, Data Vault, and feature store design.
• Experience with orchestration tools such as Airflow, Databricks Workflows, or Azure Data Factory.
• Proven experience enabling machine learning workloads at enterprise scale.
• Experience building feature pipelines, model training pipelines, and model deployment frameworks.
• Strong understanding of MLOps practices, including:
• Automated retraining
• CI/CD for ML
• Experience partnering with Data Scientists to productionise machine learning models.
• Strong experience with CI/CD tooling such as GitHub Actions, Azure DevOps, or Jenkins.
• Expertise in Infrastructure as Code (Terraform, ARM/Bicep, or CloudFormation).
• Strong understanding of data governance, metadata management, security, and compliance controls.
• Advanced programming skills in Python and SQL.
• Experience working with REST APIs, event streaming technologies (Kafka, Kinesis), and microservices architectures.

Highly Regarded

• Experience leading engineering teams across multiple regions and geographies.
• Experience with real-time analytics and streaming AI/ML workloads.
• Exposure to advanced optimisation techniques and modern AI engineering practices.
• Experience with BI and analytics tools such as Power BI or Tableau.
• Experience with Azure services including ADLS, Synapse, Event Hub, and Key Vault.
• Experience with data quality frameworks.
• Databricks and/or AWS cloud certifications.

Experience Required

• 10+ years' experience in Data Engineering, Software Engineering, or a related field.
• 5+ years working with Databricks or large-scale Spark platforms.
• 5+ years designing and delivering cloud-native data solutions.
• Demonstrated experience defining technical strategy and influencing enterprise architecture decisions.

Imagine working with us

We understand that work means different things to everyone... We know happy, healthy people make great teams, and great teams put more heart into each customer and patient interaction. And that ’ s why we ’ re reinventing work.

Imagine a workplace that helps you and your family thrive. Where connection, personal development and health and wellbeing are front of mind. To learn more about our benefits go to https://careers.medibank.com.au/culture/rewards-benefits/

For you, work should help you Live Better. It should bring you fulfillment and joy. And with Medibank, it could.

Inclusion and Accessibility

We believe in everyone's potential and strive to make Medibank inclusive for all because different perspectives make us better. We encourage applications from everyone, including Aboriginal and Torres Strait Islander peoples, neurodivergent candidates, LGBTQIA+ community including transgender and gender diverse candidates and candidates with a disability.

Medibank proudly recognised as Best Enterprise Organisation, 2026 AFR BOSS Best Places to Work

Job Segment: Cloud, Database, Neurology, Testing, IT Architecture, Technology, Healthcare

Copyright @2024 Medibank Private Limited. All rights reserved. ABN 47 080 890 259.