Senior Databricks Data Engineer: Azure Pipelines & Spark

OpenTalent; City of Brisbane; Hybrid; AUD 150,000 - 210,000; Full time

Job summary

OpenTalent seeks an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines on Azure Databricks for advanced analytics. You will collaborate with Data Scientists to deliver clean, structured datasets and ensure governance and security compliant with banking regulations.

The role emphasizes PySpark, Spark optimization, and Azure components (Data Factory, Synapse, Delta Lake) in an Agile setting across Brisbane, Sydney, or Melbourne, with flexible location and

Qualifications

• 2+ years hands-on experience in Azure Databricks.
• Strong proficiency in Apache Spark and PySpark.
• Experience with Azure Data Factory, Synapse, Blob Storage, and Delta Lake.
• Understanding ETL/ELT, data modeling, and data warehousing concepts.
• Proficiency in Python and/or Scala for data processing.
• Familiarity with CI/CD pipelines (Azure DevOps, Git, Terraform).
• Knowledge of data governance, security, and compliance in financial services.

Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Spark.
• Build Databricks notebooks with PySpark for data prep and enrichment.
• Collaborate with Data Scientists to deliver clean datasets.
• Implement data quality checks and monitoring.
• Integrate data from structured, semi-structured, and unstructured sources.
• Ensure data security, governance, and regulatory compliance in banking.

Skills

Azure Databricks

PySpark

Python

Scala

Agile working

Education

Bachelor’s or Master’s degree in CS/IS/Engineering

Tools

Azure Data Factory

Azure Synapse

Azure Blob Storage

Delta Lake

Git

Terraform

Azure DevOps

Job description

OpenTalent seeks an experienced Databricks Data Engineer to design, develop, and maintain scalable data pipelines on Azure Databricks for advanced analytics. You will collaborate with Data Scientists to deliver clean, structured datasets and ensure governance and security compliant with banking regulations.

The role emphasizes PySpark, Spark optimization, and Azure components (Data Factory, Synapse, Delta Lake) in an Agile setting across Brisbane, Sydney, or Melbourne, with flexible location and