Senior Databricks Data Engineer — Azure Spark Pipelines

Avanade; City of Brisbane; On-site; AUD 90,000 - 130,000; Full time

Job summary

Avanade in Brisbane, Australia is seeking an experienced Databricks Data Engineer to design, build and maintain scalable data pipelines on Azure Databricks. You will collaborate with Data Scientists to ensure clean, well-governed data for advanced analytics and data engineering initiatives.

Responsibilities include implementing ingestion pipelines with Azure Data Factory and Spark, developing PySpark notebooks for data prep and enrichment, and applying best practices in data modeling, security,

Qualifications

• Hands-on data engineering experience with Azure Databricks.
• Strong Spark/PySpark skills for distributed processing.
• Experience with Azure data tools and Delta Lake concepts.
• Proficient in Python/Scala for data processing.
• Familiarity with CI/CD using Azure DevOps or Git and Terraform.

Responsibilities

• Design and implement scalable ingestion pipelines on Azure Databricks.
• Develop PySpark notebooks for data prep, transformation, and enrichment.
• Collaborate with Data Scientists to deliver clean, timely data for analytics.

Skills

PySpark

Python

SQL

Data modeling

ETL/ELT

Tools

Azure Databricks

Azure Data Factory

Azure Synapse

Delta Lake

Terraform

Git

Azure DevOps

Job description

Avanade in Brisbane, Australia is seeking an experienced Databricks Data Engineer to design, build and maintain scalable data pipelines on Azure Databricks. You will collaborate with Data Scientists to ensure clean, well-governed data for advanced analytics and data engineering initiatives.

Responsibilities include implementing ingestion pipelines with Azure Data Factory and Spark, developing PySpark notebooks for data prep and enrichment, and applying best practices in data modeling, security,