Principal Data Engineer

Medibank; City of Melbourne; Presencial; AUD 180 000 - 260 000; Tempo integral

Resumo da oferta

Medibank is seeking a Principal Data Engineer to define and lead the technical strategy for its enterprise data platforms. You will shape Databricks-based lakehouse architectures, collaborate with architects, data scientists, and business leaders, and drive modern DataOps practices at scale.

You will mentor engineering teams, establish standards, and accelerate delivery across globally distributed squads while championing data governance and security across the platform.

Qualificações

• 10+ years in Data Engineering, Software Engineering, or a related field.
• 5+ years working with Databricks or large-scale Spark platforms.
• 5+ years delivering cloud-native data solutions.
• Proven ability to define technical strategy and influence enterprise architecture decisions.

Responsabilidades

• Lead design and delivery of scalable lakehouse architectures and data platforms using Databricks.
• Develop reusable frameworks, accelerators, and reference implementations.
• Partner with Engineers and Architects to solve complex business challenges with data-driven solutions.
• Mentor engineers and promote engineering excellence across teams.
• Communicate complex concepts to both technical and non-technical audiences.

Conhecimentos

Databricks

Unity Catalog

MLflow

Databricks SQL

PySpark

Spark SQL

Data modelling

Data Vault

Feature store design

Airflow

Databricks Workflows

Azure Data Factory

MLOps

CI/CD for ML

GitHub Actions

Azure DevOps

Jenkins

Terraform

ARM/Bicep

CloudFormation

Data governance

Metadata management

Python

SQL

REST APIs

Kafka

Kinesis

Microservices

Ferramentas

Databricks

Airflow

Azure Data Factory

GitHub Actions

Azure DevOps

Jenkins

Terraform

Descrição da oferta de emprego

It's a strange thing to say, because us humans are capable of incredible things. And at Medibank, we know our greatest potential lies in the people who work with us.

We strive to make real, fundamental change, driven by a simple purpose: to create the best health and wellbeing for all of Australia.

Shape the Future of Data at Medibank

At Medibank, we're passionate about creating the best health and wellbeing experiences for our customers and communities. As we continue to evolve our enterprise data and AI capabilities, we're looking for an exceptional Principal Data Engineer to lead the technical vision and engineering excellence of our next-generation data platforms.

This is a rare opportunity for a highly experienced data engineering leader. You will drive innovation across cloud-native data platforms, Databricks, AI/ML enablement, and DataOps practices while mentoring and uplifting engineering teams across the organisation.

The Opportunity

As our Principal Data Engineer, you will define and lead the technical strategy, architecture, standards, and delivery of Medibank's enterprise data platforms.

Operating at the intersection of strategy and hands-on engineering, you'll partner with architects, engineers, data scientists, and business leaders to design robust, scalable, and secure data solutions that power analytics, machine learning, and customer experiences across the enterprise.

You'll be a thought leader, mentor, innovator, and trusted advisor, helping engineering teams adopt modern platform practices and engineering excellence.

What You'll Be Doing

Technology Leadership

• Define and drive the technical roadmap for Medibank's Databricks platform, enterprise data pipelines, and AI/ML capabilities.
• Provide strategic technical leadership across data engineering, analytics, DataOps, and AI/ML platform initiatives.
• Evaluate emerging technologies and identify opportunities to innovate across data, cloud, and AI ecosystems.
• Act as a senior technical advisor to engineering leaders, architects, and business stakeholders.
• Establish engineering standards, frameworks, and best practices that scale across the organisation.
• Mentor and coach engineers, fostering a culture of continuous learning, collaboration, and technical excellence.

Solution Design & Delivery

• Lead the design and implementation of scalable lakehouse architectures and enterprise-grade data platforms using Databricks.
• Develop reusable frameworks, accelerators, libraries, and reference implementations to improve delivery velocity.
• Partner with Engineering Leads and Data Architects to solve complex business challenges through data-driven solutions.
• Guide teams on:
• Data modelling and architecture patterns
• Data quality frameworks
• Metadata management
• Facilitate technical workshops, platform onboarding sessions, and capability uplift across multiple delivery squads.
• Communicate complex technical concepts effectively to both technical and non-technical audiences.

Platform Excellence & Support

• Contribute to and help lead a Data Platform Centre of Excellence (CoE).
• Provide hands-on technical expertise across multiple programs and initiatives.
• Support and coordinate engineering delivery across globally distributed and offshore teams.
• Promote consistency, reliability, maintainability, and scalability across all platform capabilities.
• Champion modern DataOps and MLOps practices, including:
• CI/CD for data and machine learning
• Automated testing
• Platform observability and monitoring
• Drive continuous improvement initiatives to reduce technical debt and improve engineering productivity.
• Develop tools and automation capabilities that streamline the development and deployment of data products.
• Influence the adoption of modern cloud-native and AI-enabled engineering practices.

About You

Essential Experience & Skills

• Expert-level experience with Databricks, including:
• Unity Catalog
• MLflow
• Databricks SQL
• Advanced expertise in PySpark, Spark SQL, and distributed data processing.
• Strong experience designing and implementing modern lakehouse architectures.
• Deep knowledge of data modelling methodologies, including dimensional modelling, Data Vault, and feature store design.
• Experience with orchestration tools such as Airflow, Databricks Workflows, or Azure Data Factory.
• Proven experience enabling machine learning workloads at enterprise scale.
• Experience building feature pipelines, model training pipelines, and model deployment frameworks.
• Strong understanding of MLOps practices, including:
• Automated retraining
• CI/CD for ML
• Experience partnering with Data Scientists to productionise machine learning models.
• Strong experience with CI/CD tooling such as GitHub Actions, Azure DevOps, or Jenkins.
• Expertise in Infrastructure as Code (Terraform, ARM/Bicep, or CloudFormation).
• Strong understanding of data governance, metadata management, security, and compliance controls.
• Advanced programming skills in Python and SQL.
• Experience working with REST APIs, event streaming technologies (Kafka, Kinesis), and microservices architectures.

Highly Regarded

• Experience leading engineering teams across multiple regions and geographies.
• Experience with real-time analytics and streaming AI/ML workloads.
• Exposure to advanced optimisation techniques and modern AI engineering practices.
• Experience with BI and analytics tools such as Power BI or Tableau.
• Experience with Azure services including ADLS, Synapse, Event Hub, and Key Vault.
• Experience with data quality frameworks.
• Databricks and/or AWS cloud certifications.

Experience Required

• 10+ years' experience in Data Engineering, Software Engineering, or a related field.
• 5+ years working with Databricks or large-scale Spark platforms.
• 5+ years designing and delivering cloud-native data solutions.
• Demonstrated experience defining technical strategy and influencing enterprise architecture decisions.

Imagine working with us

We understand that work means different things to everyone... We know happy, healthy people make great teams, and great teams put more heart into each customer and patient interaction. And that’s why we're reinventing work.

Imagine a workplace that helps you and your family thrive. Where connection, personal development and health and wellbeing are front of mind. To learn more about our benefits go to https://careers.medibank.com.au/culture/rewards-benefits/

For you, work should help you Live Better. It should bring you fulfillment and joy. And with Medibank, it could.

Inclusion and Accessibility

We believe in everyone's potential and strive to make Medibank inclusive for all because different perspectives make us better. We encourage applications from everyone, including Aboriginal and Torres Strait Islander peoples, neurodivergent candidates, LGBTQIA+ community including transgender and gender diverse candidates and candidates with a disability.

Medibank proudly recognised as Best Enterprise Organisation, 2026 AFR BOSS Best Places to Work