Senior Data Engineer - Azure Databricks Platform Lead

Shift Financial; Sydney; Hybrid; AUD 140,000 - 190,000; Full time

Benefits offered by this job

Parental leave

Hybrid working

Strategy Week

Flat structure

Wellbeing

Job summary

Shift Financial is seeking a Senior Data Engineer to own the Azure Databricks platform and set standards for our data program in Australia. You’ll lead the data workstream for migrating a core lending system, mentor junior engineers, and collaborate with analysts and product teams to turn business needs into trusted data products.

You’ll design robust ELT pipelines, ensure CI/CD, environment separation, testing, monitoring, and cost efficiency, while shaping how data works across the

Qualifications

• 5+ years in data engineering and hands-on Azure Databricks experience.
• Strong Python and SQL, plus CI/CD and deploying pipelines as code.
• Solid dimensional modelling experience (star schemas, conformed dimensions).
• Experience mentoring engineers and leading technical decisions through design and code review.
• Ability to explain technical direction clearly to non-technical stakeholders and work across teams.
• Experience leading the data side of a core system migration is a strong plus.

Responsibilities

• Own the data platform roadmap and engineering standards, including CI/CD and environment management.
• Design and lead the build of robust ELT pipelines on Azure Databricks.
• Lead the data workstream for core system migration and remap pipelines and models.
• Build and extend a dimensional data model that becomes the trusted data source.
• Improve data quality, freshness and alerting to catch issues early.
• Keep platform cost-efficient through optimisation and compute strategy.
• Mentor junior data engineers through code reviews and pairing.
• Collaborate with analysts, data scientists, product managers and Engineering to deliver data products.

Skills

Azure Databricks

Python

SQL

CI/CD

Mentoring

Communication

Tools

Unity Catalog

Delta Lake

Databricks Asset Bundles

Job description

Shift Financial is seeking a Senior Data Engineer to own the Azure Databricks platform and set standards for our data program in Australia. You’ll lead the data workstream for migrating a core lending system, mentor junior engineers, and collaborate with analysts and product teams to turn business needs into trusted data products.

You’ll design robust ELT pipelines, ensure CI/CD, environment separation, testing, monitoring, and cost efficiency, while shaping how data works across the