Data Engineer

XPT Software Australia Pty Ltd; Sydney; On-site; AUD 180,000 - 230,000; Full time

Job summary

XPT Software Australia Pty Ltd is seeking a Senior Data Engineer in Sydney to design, build, and optimize data pipelines and lakehouse solutions that support analytics, reporting, and enterprise decision-making. You will work with Azure Databricks, Delta Lake, and a broad Azure data ecosystem, delivering production-grade pipelines and governance across environments.

The role emphasizes scalable batch/streaming processing, data quality, and cost-aware Spark tuning while collaborating with

Qualifications

• 8+ years of experience in data engineering, data integration, ETL/ELT, or large-scale data platform delivery.
• Strong hands-on experience with Azure Databricks, notebooks, jobs or workflows, cluster configuration, debugging, and performance tuning.
• Advanced proficiency in Python, PySpark, and SQL for production-grade data engineering.
• Strong experience with Delta Lake and lakehouse architecture, including reliable incremental processing and data-quality controls.
• Practical experience with Azure Data Lake Storage Gen2 and Azure Data Factory.
• Experience implementing CI/CD using Azure DevOps, Git, branching strategies, automated testing, and environment-based deployment controls.
• Strong understanding of dimensional modelling, data warehousing, data lake design, metadata, lineage, partitioning, and file formats such as Parquet.

Responsibilities

• Design, build, and optimize scalable batch and streaming data pipelines using Azure Databricks, PySpark, Python, SQL, and Delta Lake.
• Develop reusable ingestion, transformation, validation, and publishing frameworks for large-scale enterprise datasets.
• Implement medallion or layered lakehouse patterns across raw, curated, and consumption-ready data zones.
• Integrate Azure Databricks with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Key Vault, Azure DevOps, and related Azure services.
• Develop production-grade Databricks notebooks, jobs, workflows, libraries, and cluster configurations.
• Create complex SQL and PySpark transformations, data-quality checks, reconciliation logic, and error-handling routines.
• Optimize Spark workloads, storage layouts, partitioning, file sizes, caching, and query performance while controlling platform cost.
• Implement Delta Lake capabilities such as ACID transactions, schema enforcement, schema evolution, time travel, and merge operations.
• Build automated CI/CD pipelines for code promotion, configuration management, testing, and deployment across environments.
• Apply data security and governance controls using role-based access, secrets management, data masking, auditability, and controlled access patterns.
• Monitor data pipelines, troubleshoot failures, perform root-cause analysis, and deliver sustainable production fixes.
• Partner with Architects, Business Analysts, Data Analysts, Product Owners, and business stakeholders to convert requirements into reliable data products.
• Provide technical leadership, code reviews, engineering standards, and mentoring to other data engineers.

Skills

Azure Databricks

Python

PySpark

SQL

Delta Lake

Azure Data Lake Storage Gen2

Azure Data Factory

Azure DevOps

CI/CD

Data governance

Data security

ETL/ELT

Tools

Terraform

Bicep

dbt

Airflow

Kafka

Power BI

Job description

XPT Software Australia Pty Ltd | Contract

Sydney, Australia | Posted on 10/30/2026

• XPT SoftwareAustralia PTY Ltd, incorporated in 2016, is a Software Services company
  
• XPT works with topclients across Australia in Banking, Insurance, Telecom,Retail, Energy, Mining and Manufacturingdomains.
  
• We have 120+technocrats in Australia working at our clientlocations.
  
• XPT SoftwareAustralia is part of group companies which has globalpresence across India & Europe.
  
• We have served100+ clients globally, fulfilling their onsite-offshoreneeds.
  

Job Description

Role Overview

We are looking for a Senior Data Engineerwith strong hands-on experience in Azure Databricks and the Microsoft Azuredata ecosystem. The role will design, build, optimise, and support scalabledata pipelines and lakehouse solutions that enable analytics, reporting, datascience, and enterprise decision-making.

What You'll Work On

• Design,build, and optimise scalable batch and streaming data pipelines using AzureDatabricks, PySpark, Python, SQL, and Delta Lake.
• Developreusable ingestion, transformation, validation, and publishing frameworks forlarge-scale enterprise datasets.
• Implementmedallion or layered lakehouse patterns across raw, curated, andconsumption-ready data zones.
• IntegrateAzure Databricks with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Key Vault, Azure DevOps, and related Azure services.
• Developproduction-grade Databricks notebooks, jobs, workflows, libraries, and clusterconfigurations.
• Createcomplex SQL and PySpark transformations, data-quality checks, reconciliationlogic, and error-handling routines.
• OptimiseSpark workloads, storage layouts, partitioning, file sizes, caching, and queryperformance while controlling platform cost.
• ImplementDelta Lake capabilities such as ACID transactions, schema enforcement, schemaevolution, time travel, and merge operations.
• Buildautomated CI/CD pipelines for code promotion, configuration management,testing, and deployment across environments.
• Applydata security and governance controls using role-based access, secretsmanagement, data masking, auditability, and controlled access patterns.
• Monitordata pipelines, troubleshoot failures, perform root-cause analysis, and deliversustainable production fixes.
• Partnerwith Architects, Business Analysts, Data Analysts, Product Owners, and businessstakeholders to convert requirements into reliable data products.
• Providetechnical leadership, code reviews, engineering standards, and mentoring toother data engineers.

KeySkills and Experience

• 8+years of experience in data engineering, data integration, ETL/ELT, orlarge-scale data platform delivery.
• Stronghands-on experience with Azure Databricks, including notebooks, jobs orworkflows, cluster configuration, debugging, and performance tuning.
• Advancedproficiency in Python, PySpark, and SQL for production-grade data engineering.
• Strongexperience with Delta Lake and lakehouse architecture, including reliableincremental processing and data-quality controls.
• Practicalexperience with Azure Data Lake Storage Gen2 and Azure Data Factory.
• Experienceimplementing CI/CD using Azure DevOps, Git, branching strategies, automatedtesting, and environment-based deployment controls.
• Strongunderstanding of dimensional modelling, data warehousing, data lake design,metadata, lineage, partitioning, and file formats such as Parquet.
• Experiencewith orchestration, monitoring, alerting, operational support, incidentanalysis, and service-level expectations.
• Understandingof Azure identity, Key Vault, managed identities, networking, privateendpoints, and secure data-platform patterns.
• Strongcommunication, documentation, problem-solving, code-review, and stakeholdercollaboration skills.

Good to Have

• Experiencewith Microsoft Purview, Unity Catalog, Azure Synapse Analytics, Event Hubs, orStructured Streaming.
• Knowledgeof infrastructure as code using Terraform or Bicep.
• Experiencewith dbt, Airflow, Kafka, Power BI, or machine-learning data pipelines.
• Experiencein telecommunications or another high-volume enterprise environment.