Senior Data Engineer - Azure Databricks & Lakehouse Lead

XPT Software; Sydney; On-site; AUD 140,000 - 190,000; Full time

Job summary

XPT Software is seeking a Senior Data Engineer in Sydney, Australia, to design, build, and support scalable data pipelines and lakehouse solutions using Azure Databricks and the Microsoft Azure data ecosystem.

You will design batch and streaming data pipelines, implement lakehouse patterns, integrate with Azure Data Lake Gen2, Data Factory, Key Vault, and DevOps, and lead production-grade Databricks notebooks, jobs, and CI/CD workflows.

Qualifications

• 8+ years in data engineering or large-scale data platforms.
• Hands-on with Azure Databricks, PySpark, Python and SQL.
• Experience with Delta Lake, lakehouse patterns, and data lake design.
• CI/CD using Azure DevOps, Git and automated testing.
• Knowledge of data governance, security and metadata concepts.

Responsibilities

• Design, build and optimize scalable batch and streaming pipelines.
• Develop ingestion, transformation, validation, and publishing frameworks.
• Implement lakehouse patterns across data zones.
• Integrate Databricks with Azure services and manage notebooks and workflows.
• Create data quality checks, lineage, and monitoring; optimize performance.
• Lead code reviews, set standards, and mentor other engineers.

Tools

Azure Databricks

PySpark

Python

SQL

Delta Lake

Azure Data Lake Storage Gen2

Azure Data Factory

Azure DevOps

Git

Parquet

Spark

Terraform

Bicep

Job description

XPT Software is seeking a Senior Data Engineer in Sydney, Australia, to design, build, and support scalable data pipelines and lakehouse solutions using Azure Databricks and the Microsoft Azure data ecosystem.

You will design batch and streaming data pipelines, implement lakehouse patterns, integrate with Azure Data Lake Gen2, Data Factory, Key Vault, and DevOps, and lead production-grade Databricks notebooks, jobs, and CI/CD workflows.