Data Engineer

XPT Software

Sydney

On-site

AUD 140,000 - 190,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

XPT Software is seeking a Senior Data Engineer in Sydney, Australia, to design, build, and support scalable data pipelines and lakehouse solutions using Azure Databricks and the Microsoft Azure data ecosystem.

You will design batch and streaming data pipelines, implement lakehouse patterns, integrate with Azure Data Lake Gen2, Data Factory, Key Vault, and DevOps, and lead production-grade Databricks notebooks, jobs, and CI/CD workflows.

Qualifications

  • 8+ years in data engineering or large-scale data platforms.
  • Hands-on with Azure Databricks, PySpark, Python and SQL.
  • Experience with Delta Lake, lakehouse patterns, and data lake design.
  • CI/CD using Azure DevOps, Git and automated testing.
  • Knowledge of data governance, security and metadata concepts.

Responsibilities

  • Design, build and optimize scalable batch and streaming pipelines.
  • Develop ingestion, transformation, validation, and publishing frameworks.
  • Implement lakehouse patterns across data zones.
  • Integrate Databricks with Azure services and manage notebooks and workflows.
  • Create data quality checks, lineage, and monitoring; optimize performance.
  • Lead code reviews, set standards, and mentor other engineers.

Tools

Azure Databricks
PySpark
Python
SQL
Delta Lake
Azure Data Lake Storage Gen2
Azure Data Factory
Azure DevOps
Git
Parquet
Spark
Terraform
Bicep

Job description

We are looking for a Senior Data Engineer with strong hands-on experience in Azure Databricks and the Microsoft Azure data ecosystem. The role will design, build, optimise, and support scalable data pipelines and lakehouse solutions that enable analytics, reporting, data science, and enterprise decision-making.

What You'll Work On
  • Design, build, and optimise scalable batch and streaming data pipelines using Azure Databricks, PySpark, Python, SQL, and Delta Lake.
  • Develop reusable ingestion, transformation, validation, and publishing frameworks for large-scale enterprise datasets.
  • Implement medallion or layered lakehouse patterns across raw, curated, and consumption-ready data zones.
  • Integrate Azure Databricks with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Key Vault, Azure DevOps, and related Azure services.
  • Develop production-grade Databricks notebooks, jobs, workflows, libraries, and cluster configurations.
  • Create complex SQL and PySpark transformations, data-quality checks, reconciliation logic, and error-handling routines.
  • Optimise Spark workloads, storage layouts, partitioning, file sizes, caching, and query performance while controlling platform cost.
  • Implement Delta Lake capabilities such as ACID transactions, schema enforcement, schema evolution, time travel, and merge operations.
  • Build automated CI/CD pipelines for code promotion, configuration management, testing, and deployment across environments.
  • Apply data security and governance controls using role-based access, secrets management, data masking, auditability, and controlled access patterns.
  • Monitor data pipelines, troubleshoot failures, perform root-cause analysis, and deliver sustainable production fixes.
  • Partner with Architects, Business Analysts, Data Analysts, Product Owners, and business stakeholders to convert requirements into reliable data products.
  • Provide technical leadership, code reviews, engineering standards, and mentoring to other data engineers.
Key Skills and Experience
  • 8+ years of experience in data engineering, data integration, ETL/ELT, or large-scale data platform delivery.
  • Strong hands-on experience with Azure Databricks, including notebooks, jobs or workflows, cluster configuration, debugging, and performance tuning.
  • Advanced proficiency in Python, PySpark, and SQL for production-grade data engineering.
  • Strong experience with Delta Lake and lakehouse architecture, including reliable incremental processing and data-quality controls.
  • Practical experience with Azure Data Lake Storage Gen2 and Azure Data Factory.
  • Experience implementing CI/CD using Azure DevOps, Git, branching strategies, automated testing, and environment-based deployment controls.
  • Strong understanding of dimensional modelling, data warehousing, data lake design, metadata, lineage, partitioning, and file formats such as Parquet.
  • Experience with orchestration, monitoring, alerting, operational support, incident analysis, and service-level expectations.
  • Understanding of Azure identity, Key Vault, managed identities, networking, private endpoints, and secure data-platform patterns.
  • Strong communication, documentation, problem-solving, code-review, and stakeholder collaboration skills.
Good to Have
  • Experience with Microsoft Purview, Unity Catalog, Azure Synapse Analytics, Event Hubs, or Structured Streaming.
  • Knowledge of infrastructure as code using Terraform or Bicep.
  • Experience with dbt, Airflow, Kafka, Power BI, or machine-learning data pipelines.
  • Experience in telecommunications or another high-volume enterprise environment.
  • Relevant Microsoft Azure or Databricks certification.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

XPT Software • Canberra

On-site
AUD 150,000 - 190,000
Data Engineer
Data Engineer

XPT Software Australia Pty Ltd • Sydney

On-site
AUD 180,000 - 230,000
Senior Data Engineer
Senior Data Engineer

DX1 • Melbourne

On-site
AUD 160,000 - 230,000
Senior Data Engineer
Senior Data Engineer

Xpt Software Australia Pty • New South Wales

On-site
AUD 140,000 - 180,000
Senior Azure Data Engineer – Databricks -Sydney Only
Senior Azure Data Engineer – Databricks -Sydney Only

The Hr Ally • Sydney

On-site
AUD 180,000 - 240,000
Data Engineer
Data Engineer

Mane Consulting • Canberra

On-site
AUD 110,000 - 150,000
Senior Data Engineer
Senior Data Engineer

Coforge Limited • Brisbane

On-site
AUD 140,000 - 190,000
Azure Data Engineer
Azure Data Engineer

Tech Mahindra Limited • Sydney

On-site
AUD 130,000 - 180,000
Azure data engineer
Azure data engineer

Spait Infotech • Victoria

On-site
AUD 95,000 - 130,000
Databricks - Practice Lead / Data & Analytics Consulting
Databricks - Practice Lead / Data & Analytics Consulting

Find My New Job • Melbourne

On-site
AUD 180,000 - 260,000
Top locations