Data Engineer

XPT Software; Canberra; On-site; AUD 150,000 - 190,000; Full time

Job summary

XPT Software in Canberra, ACT, is seeking a Senior Data Engineer with strong hands-on experience in Azure Databricks to design, build, and optimize scalable data pipelines and lakehouse solutions for analytics and decision-making.

You will implement batch and streaming pipelines, integrate with Azure Data Lake Storage Gen2, Data Factory, and Key Vault, and deliver production-grade notebooks, jobs, and CI/CD workflows while enforcing data governance and cost efficiency.

Qualifications

• 8+ years of experience in data engineering or data platform delivery.
• Hands-on Azure Databricks expertise with notebooks, jobs, and workflows.
• Advanced Python, PySpark, and SQL proficiency for production pipelines.
• Experience with Delta Lake and lakehouse architectures.
• Experience with Azure Data Lake Storage Gen2 and Azure Data Factory.
• CI/CD in Azure DevOps and version control with Git.
• Knowledge of dimensional modeling, metadata, and data governance.
• Strong communication and collaboration skills.

Responsibilities

• Design, build, and optimize scalable batch and streaming data pipelines on Azure Databricks.
• Develop reusable ingestion, transformation, validation, and publishing frameworks.
• Implement Delta Lake features and ACID transactions.
• Create production-grade notebooks, jobs, and cluster configurations.
• Monitor pipelines, troubleshoot issues, and perform root-cause analysis.
• Partner with Architects, Analysts, and stakeholders to translate requirements into data products.
• Lead code reviews, establish engineering standards, and mentor peers.
• Build CI/CD pipelines for code promotion and deployment across environments.

Skills

Data engineering experience

Cloud data architecture

Strong communication

Problem solving

Code reviews

Stakeholder collaboration

Tools

Azure Databricks

Azure Data Lake Storage Gen2

Azure Data Factory

Delta Lake

Azure DevOps

PySpark

Python

SQL

Job description

• XPT SoftwareAustralia PTY Ltd, incorporated in 2016, is a Software Services company
• XPT works with topclients across Australia in Banking, Insurance, Telecom,Retail, Energy, Mining and Manufacturingdomains.
• We have 120+technocrats in Australia working at our clientlocations.
• XPT SoftwareAustralia is part of group companies which has globalpresence across India & Europe.
• We have served100+ clients globally, fulfilling their onsite-offshoreneeds.

Role Overview

We are looking for a Senior Data Engineerwith strong hands-on experience in Azure Databricks and the Microsoft Azuredata ecosystem. The role will design, build, optimise, and support scalabledata pipelines and lakehouse solutions that enable analytics, reporting, datascience, and enterprise decision-making.

What You'll Work On

• Design,build, and optimise scalable batch and streaming data pipelines using AzureDatabricks, PySpark, Python, SQL, and Delta Lake.
• Developreusable ingestion, transformation, validation, and publishing frameworks forlarge-scale enterprise datasets.
• Implementmedallion or layered lakehouse patterns across raw, curated, andconsumption-ready data zones.
• IntegrateAzure Databricks with Azure Data Lake Storage Gen2, Azure Data Factory, AzureKey Vault, Azure DevOps, and related Azure services.
• Developproduction-grade Databricks notebooks, jobs, workflows, libraries, and clusterconfigurations.
• Createcomplex SQL and PySpark transformations, data-quality checks, reconciliationlogic, and error-handling routines.
• OptimiseSpark workloads, storage layouts, partitioning, file sizes, caching, and queryperformance while controlling platform cost.
• ImplementDelta Lake capabilities such as ACID transactions, schema enforcement, schemaevolution, time travel, and merge operations.
• Buildautomated CI/CD pipelines for code promotion, configuration management,testing, and deployment across environments.
• Applydata security and governance controls using role-based access, secretsmanagement, data masking, auditability, and controlled access patterns.
• Monitordata pipelines, troubleshoot failures, perform root-cause analysis, and deliversustainable production fixes.
• Partnerwith Architects, Business Analysts, Data Analysts, Product Owners, and businessstakeholders to convert requirements into reliable data products.
• Providetechnical leadership, code reviews, engineering standards, and mentoring toother data engineers.

Key Skills and Experience

• 8+years of experience in data engineering, data integration, ETL/ELT, orlarge-scale data platform delivery.
• Stronghands-on experience with Azure Databricks, including notebooks, jobs orworkflows, cluster configuration, debugging, and performance tuning.
• Advancedproficiency in Python, PySpark, and SQL for production-grade data engineering.
• Strongexperience with Delta Lake and lakehouse architecture, including reliableincremental processing and data-quality controls.
• Practicalexperience with Azure Data Lake Storage Gen2 and Azure Data Factory.
• Experienceimplementing CI/CD using Azure DevOps, Git, branching strategies, automatedtesting, and environment-based deployment controls.
• Strongunderstanding of dimensional modelling, data warehousing, data lake design,metadata, lineage, partitioning, and file formats such as Parquet.
• Experiencewith orchestration, monitoring, alerting, operational support, incidentanalysis, and service-level expectations.
• Understandingof Azure identity, Key Vault, managed identities, networking, privateendpoints, and secure data-platform patterns.
• Strongcommunication, documentation, problem-solving, code-review, and stakeholdercollaboration skills.

Good to Have

• Experiencewith Microsoft Purview, Unity Catalog, Azure Synapse Analytics, Event Hubs, orStructured Streaming.
• Knowledgeof infrastructure as code using Terraform or Bicep.
• Experiencewith dbt, Airflow, Kafka, Power BI, or machine-learning data pipelines.
• Experiencein telecommunications or another high-volume enterprise environment.