Senior Data Engineer - Azure Databricks Lakehouse

XPT Software; Sydney; On-site; AUD 150,000 - 190,000; Full time

Job summary

XPT Software in Sydney is seeking a Senior Data Engineer to design, build, and optimise scalable data pipelines and lakehouse solutions using Azure Databricks and the wider Microsoft Azure data stack.

You will develop production-grade pipelines, implement lakehouse patterns, manage data quality, and collaborate with architects and stakeholders to deliver reliable data products at scale.

Qualifications

• 8+ years of experience in data engineering, data integration, ETL/ELT, or large-scale data platform delivery.

Responsibilities

• Design, build, and optimise scalable batch and streaming data pipelines using Azure Databricks, PySpark, Python, SQL, and Delta Lake.
• Develop reusable ingestion, transformation, validation, and publishing frameworks for large-scale enterprise datasets.
• Implement medallion or layered lakehouse patterns across raw, curated, and consumption-ready data zones.
• Integrate Azure Databricks with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Key Vault, Azure DevOps, and related Azure services.

Skills

Azure Databricks

Python

PySpark

SQL

Delta Lake

Azure Data Lake Gen2

Azure Data Factory

Azure DevOps

Dimensional Modeling

Data Warehousing

Data Lake Design

Metadata & Lineage

Partitioning

Parquet

CI/CD

Tools

Terraform

Bicep

dbt

Airflow

Kafka

Power BI

Job description

XPT Software in Sydney is seeking a Senior Data Engineer to design, build, and optimise scalable data pipelines and lakehouse solutions using Azure Databricks and the wider Microsoft Azure data stack.

You will develop production-grade pipelines, implement lakehouse patterns, manage data quality, and collaborate with architects and stakeholders to deliver reliable data products at scale.