Azure Databricks Data Engineer: Build Scalable Pipelines

Avanade Inc.; City of Melbourne; On-site; AUD 120,000 - 180,000; Full time

Job summary

Avanade Inc. is seeking an experienced Databricks Data Engineer Consultant to design, develop, and maintain scalable data pipelines on Azure Databricks, enabling advanced analytics and data engineering initiatives.

You will collaborate with Data Scientists to understand data requirements, implement data quality checks, and optimize Spark jobs for performance and cost on Azure, while upholding data security and governance standards in regulated banking contexts.

Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.
• Build and maintain Databricks notebooks using PySpark for data preparation and transformation.
• Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets.
• Implement data quality checks, validation rules, and monitoring mechanisms.
• Integrate data from various sources including structured, semi-structured, and unstructured formats.
• Ensure data security, governance, and compliance with banking regulations.
• Optimize Spark jobs for performance and cost-efficiency in Azure.
• Participate in code reviews, design discussions, and agile ceremonies.
• Document data pipelines, workflows, and technical decisions.

Skills

Databricks

PySpark

Azure Data Factory

Apache Spark

Tools

Azure Databricks

Azure Data Factory

Job description

Avanade Inc. is seeking an experienced Databricks Data Engineer Consultant to design, develop, and maintain scalable data pipelines on Azure Databricks, enabling advanced analytics and data engineering initiatives.

You will collaborate with Data Scientists to understand data requirements, implement data quality checks, and optimize Spark jobs for performance and cost on Azure, while upholding data security and governance standards in regulated banking contexts.