Principal Data Engineer

Medibank; City of Melbourne; On-site; AUD 210,000 - 260,000; Full time

Job summary

Medibank is seeking a Principal Data Engineer to shape the technical strategy and lead Databricks-based data platforms across the enterprise. This role drives innovation in cloud-native data platforms, AI/ML enablement, and DataOps practices while mentoring engineering teams.

You will partner with architects, engineers, data scientists and business leaders to design robust, scalable data solutions that power analytics, machine learning and customer experiences across the organisation.

Qualifications

• Expert Databricks experience with Unity Catalog, MLflow and Databricks SQL.
• Proficiency in PySpark, Spark SQL and distributed data processing.
• Strong data modeling knowledge (dimensional, Data Vault, feature store).
• Experience with orchestration tools (Airflow, Databricks Workflows, Azure Data Factory).
• Proven ability to enable enterprise-scale ML workloads and data pipelines.

Responsibilities

• Define and drive the technical roadmap for Databricks platform and AI capabilities.
• Lead the design of scalable lakehouse architectures and data platforms.
• Mentor engineers and promote DataOps and MLOps practices across teams.
• Partner with architects, data scientists and business leaders to deliver data-driven solutions.
• Champion governance, security and compliance across platforms.

Skills

Databricks

PySpark

Data Modeling

Data Governance

MLOps

ML CI/CD

Python

SQL

REST APIs

Kafka

Microservices

Terraform

Tools

Unity Catalog

MLflow

Databricks SQL

Airflow

Azure Data Factory

Terraform

GitHub Actions

Azure DevOps

Job description

It’s a strange thing to say, because us humans are capable of incredible things. And at Medibank, we know our greatest potential lies in the people who work with us.

We strive to make real, fundamental change, driven by a simple purpose: to create the best health and wellbeing for all of Australia.

Shape the Future of Data at Medibank

At Medibank, we're passionate about creating the best health and wellbeing experiences for our customers and communities. As we continue to evolve our enterprise data and AI capabilities, we're looking for an exceptional Principal Data Engineer to lead the technical vision and engineering excellence of our next-generation data platforms.

This is a rare opportunity for a highly experienced data engineering leader. You will drive innovation across cloud-native data platforms, Databricks, AI/ML enablement, and DataOps practices while mentoring and uplifting engineering teams across the organisation.

The Opportunity

As our Principal Data Engineer, you will define and lead the technical strategy, architecture, standards, and delivery of Medibank's enterprise data platforms.

Operating at the intersection of strategy and hands-on engineering, you'll partner with architects, engineers, data scientists, and business leaders to design robust, scalable, and secure data solutions that power analytics, machine learning, and customer experiences across the enterprise.

You'll be a thought leader, mentor, innovator, and trusted advisor, helping engineering teams adopt modern platform practices and engineering excellence.

What You'll Be Doing

Technology Leadership

• Define and drive the technical roadmap for Medibank's Databricks platform, enterprise data pipelines, and AI/ML capabilities.
• Provide strategic technical leadership across data engineering, analytics, DataOps, and AI/ML platform initiatives.
• Evaluate emerging technologies and identify opportunities to innovate across data, cloud, and AI ecosystems.
• Act as a senior technical advisor to engineering leaders, architects, and business stakeholders.
• Establish engineering standards, frameworks, and best practices that scale across the organisation.
• Mentor and coach engineers, fostering a culture of continuous learning, collaboration, and technical excellence.

Solution Design & Delivery

• Lead the design and implementation of scalable lakehouse architectures and enterprise-grade data platforms using Databricks.
• Develop reusable frameworks, accelerators, libraries, and reference implementations to improve delivery velocity.
• Partner with Engineering Leads and Data Architects to solve complex business challenges through data-driven solutions.
• Guide teams on:
• Data modelling and architecture patterns
• Data quality frameworks
• Metadata management
• Facilitate technical workshops, platform onboarding sessions, and capability uplift across multiple delivery squads.
• Communicate complex technical concepts effectively to both technical and non-technical audiences.

Platform Excellence & Support

• Contribute to and help lead a Data Platform Centre of Excellence (CoE).
• Provide hands-on technical expertise across multiple programs and initiatives.
• Support and coordinate engineering delivery across globally distributed and offshore teams.
• Promote consistency, reliability, maintainability, and scalability across all platform capabilities.
• Champion modern DataOps and MLOps practices, including:
• CI/CD for data and machine learning
• Automated testing
• Platform observability and monitoring
• Drive continuous improvement initiatives to reduce technical debt and improve engineering productivity.
• Develop tools and automation capabilities that streamline the development and deployment of data products.
• Influence the adoption of modern cloud-native and AI-enabled engineering practices.

About You

Essential Experience & Skills

• Expert-level experience with Databricks, including:
• Unity Catalog
• MLflow
• Databricks SQL
• Advanced expertise in PySpark, Spark SQL, and distributed data processing.
• Strong experience designing and implementing modern lakehouse architectures.
• Deep knowledge of data modelling methodologies, including dimensional modelling, Data Vault, and feature store design.
• Experience with orchestration tools such as Airflow, Databricks Workflows, or Azure Data Factory.
• Proven experience enabling machine learning workloads at enterprise scale.
• Experience building feature pipelines, model training pipelines, and model deployment frameworks.
• Strong understanding of MLOps practices, including:
• Automated retraining
• CI/CD for ML
• Experience partnering with Data Scientists to productionise machine learning models.
• Strong experience with CI/CD tooling such as GitHub Actions, Azure DevOps, or Jenkins.
• Expertise in Infrastructure as Code (Terraform, ARM/Bicep, or CloudFormation).
• Strong understanding of data governance, metadata management, security, and compliance controls.
• Advanced programming skills in Python and SQL.
• Experience working with REST APIs, event streaming technologies (Kafka, Kinesis), and microservices architectures.

Highly Regarded

• Experience leading engineering teams across multiple regions and geographies.
• Experience with real-time analytics and streaming AI/ML workloads.
• Exposure to advanced optimisation techniques and modern AI engineering practices.
• Experience with BI and analytics tools such as Power BI or Tableau.
• Experience with Azure services including ADLS, Synapse, Event Hub, and Key Vault.
• Experience with data quality frameworks.
• Databricks and/or AWS cloud certifications.

Experience Required

• 10+ years' experience in Data Engineering, Software Engineering, or a related field.
• 5+ years working with Databricks or large-scale Spark platforms.
• 5+ years designing and delivering cloud-native data solutions.
• Demonstrated experience defining technical strategy and influencing enterprise architecture decisions.

Imagine working with us

We understand that work means different things to everyone... We know happy, healthy people make great teams, and great teams put more heart into each customer and patient interaction. And that’s why we’re reinventing work.

Imagine a workplace that helps you and your family thrive. Where connection, personal development and health and wellbeing are front of mind. To learn more about our benefits go to https://careers.medibank.com.au/culture/rewards-benefits/

For you, work should help you Live Better. It should bring you fulfillment and joy. And with Medibank, it could.

Inclusion and Accessibility

We believe in everyone's potential and strive to make Medibank inclusive for all because different perspectives make us better. We encourage applications from everyone, including Aboriginal and Torres Strait Islander peoples, neurodivergent candidates, LGBTQIA+ community including transgender and gender diverse candidates and candidates with a disability.

Medibank proudly recognised as Best Enterprise Organisation, 2026 AFR BOSS Best Places to Work