Lead Data Platform Architect & AI Engineer

Medibank; City of Melbourne; On-site; AUD 210,000 - 260,000; Full time

Job summary

Medibank is seeking a Principal Data Engineer to shape the technical strategy and lead Databricks-based data platforms across the enterprise. This role drives innovation in cloud-native data platforms, AI/ML enablement, and DataOps practices while mentoring engineering teams.

You will partner with architects, engineers, data scientists and business leaders to design robust, scalable data solutions that power analytics, machine learning and customer experiences across the organisation.

Qualifications

• Expert Databricks experience with Unity Catalog, MLflow and Databricks SQL.
• Proficiency in PySpark, Spark SQL and distributed data processing.
• Strong data modeling knowledge (dimensional, Data Vault, feature store).
• Experience with orchestration tools (Airflow, Databricks Workflows, Azure Data Factory).
• Proven ability to enable enterprise-scale ML workloads and data pipelines.

Responsibilities

• Define and drive the technical roadmap for Databricks platform and AI capabilities.
• Lead the design of scalable lakehouse architectures and data platforms.
• Mentor engineers and promote DataOps and MLOps practices across teams.
• Partner with architects, data scientists and business leaders to deliver data-driven solutions.
• Champion governance, security and compliance across platforms.

Skills

Databricks

PySpark

Data Modeling

Data Governance

MLOps

ML CI/CD

Python

SQL

REST APIs

Kafka

Microservices

Terraform

Tools

Unity Catalog

MLflow

Databricks SQL

Airflow

Azure Data Factory

Terraform

GitHub Actions

Azure DevOps

Job description

Medibank is seeking a Principal Data Engineer to shape the technical strategy and lead Databricks-based data platforms across the enterprise. This role drives innovation in cloud-native data platforms, AI/ML enablement, and DataOps practices while mentoring engineering teams.

You will partner with architects, engineers, data scientists and business leaders to design robust, scalable data solutions that power analytics, machine learning and customer experiences across the organisation.