Senior Azure Data Engineer – Databricks -Sydney Only

The Hr Ally; Sydney; Presencial; AUD 180.000 - 240.000; Jornada completa

Descripción de la vacante

The Hr Ally is seeking a Senior Azure Data Engineer / Lead in Sydney, Australia with 8+ years of software and data engineering experience. You will design, build and optimize large-scale data pipelines using PySpark, Databricks, Azure Synapse, and ADF, while collaborating with cross-functional teams in an Agile environment.

The role requires strong Python/SQL skills, data warehousing expertise, and familiarity with CI/CD and MLOps practices.

Formación

• 8+ years of software/data engineering and development.
• Strong hands-on Python and SQL; Java is a bonus.
• Design, develop and optimize complex data pipelines and ETL workflows.
• Hands-on PySpark, Databricks, Azure Synapse, Azure Data Factory (ADF) and Azure Functions.
• Data modelling, data warehousing and large-scale ETL architectures.
• Experience with MS SQL, ADLS Gen2 and NoSQL/data storage tech.
• Knowledge of object storage and Big Data architectures, especially ADLS Gen2.
• Experience managing distributed systems and clusters for large-scale processing.
• Knowledge of MLOps architecture and practices.
• Modern data modelling concepts and enterprise data warehouse architecture.
• Familiarity with Azure DevOps, Git and CI/CD practices.
• ETL and data warehousing methodologies.
• Informatica is a plus.
• Collaborate with architects, developers and stakeholders in Agile environments.

Responsabilidades

• Design and implement end-to-end data pipelines and ETL workflows.
• Develop complex data platforms in Azure with Databricks and Spark.
• Optimize performance and cost of distributed data processing.
• Model data and maintain data warehouses for analytics.
• Collaborate with data teams and stakeholders in Agile teams.
• Ensure CI/CD and version control for data pipelines.

Conocimientos

Python

SQL

Spark

Databricks

Azure

ETL

Data pipelines

MLOps

CI/CD

Agile

Herramientas

Azure Synapse

Azure Data Factory

Azure Functions

MS SQL

ADLS Gen2

NoSQL

Descripción del empleo

Senior Azure Data Engineer-Databricks

Experience:8+Years

Technology Focus: Azure | Databricks | PySpark | Python | SQL | Data Engineering

Role Overview

We are looking for an experienced Senior Azure Data Engineer / Lead with 8+ years of software/data engineering experience and strong expertise in Azure, Databricks, Python, SQL and Apache Spark.

The ideal candidate will have hands‑on experience designing and developing complex data pipelines, distributed data processing solutions, ETL frameworks and large‑scale data platforms within Agile environments.

Key Responsibilities & Requirements

• 8+ years of experience in software/data engineering and development, preferably within Agile environments.
• Strong hands‑on expertise in Python and SQL; Java experience is an advantage.
• Design, develop and optimize complex data pipelines and ETL workflows.
• Strong experience with Apache PySpark, Databricks, Azure Synapse, Azure Data Factory (ADF) and Azure Functions.
• Hands‑on experience with data modelling, data warehousing and large‑scale ETL architectures.
• Experience working with MS SQL, ADLS Gen2 and NoSQL/data storage technologies.
• Good understanding of object storage and Big Data architectures, particularly Azure Data Lake Storage Gen2.
• Experience managing and optimizing distributed systems and clusters for large‑scale/batch data processing.
• Knowledge of MLOps architecture and practices.
• Strong understanding of modern data modelling concepts and enterprise data warehouse architecture.
• Familiarity with Azure DevOps, Git and modern CI/CD practices.
• Good knowledge of ETL and data warehousing methodologies.
• Informatica experience is an advantage.
• Ability to work closely with architects, developers, data teams and business stakeholders in an Agile environment.