Senior Azure Data Engineer | Databricks & PySpark Lead

The Hr Ally; Sydney; Presencial; AUD 180.000 - 240.000; Jornada completa

Descripción de la vacante

The Hr Ally is seeking a Senior Azure Data Engineer / Lead in Sydney, Australia with 8+ years of software and data engineering experience. You will design, build and optimize large-scale data pipelines using PySpark, Databricks, Azure Synapse, and ADF, while collaborating with cross-functional teams in an Agile environment.

The role requires strong Python/SQL skills, data warehousing expertise, and familiarity with CI/CD and MLOps practices.

Formación

• 8+ years of software/data engineering and development.
• Strong hands-on Python and SQL; Java is a bonus.
• Design, develop and optimize complex data pipelines and ETL workflows.
• Hands-on PySpark, Databricks, Azure Synapse, Azure Data Factory (ADF) and Azure Functions.
• Data modelling, data warehousing and large-scale ETL architectures.
• Experience with MS SQL, ADLS Gen2 and NoSQL/data storage tech.
• Knowledge of object storage and Big Data architectures, especially ADLS Gen2.
• Experience managing distributed systems and clusters for large-scale processing.
• Knowledge of MLOps architecture and practices.
• Modern data modelling concepts and enterprise data warehouse architecture.
• Familiarity with Azure DevOps, Git and CI/CD practices.
• ETL and data warehousing methodologies.
• Informatica is a plus.
• Collaborate with architects, developers and stakeholders in Agile environments.

Responsabilidades

• Design and implement end-to-end data pipelines and ETL workflows.
• Develop complex data platforms in Azure with Databricks and Spark.
• Optimize performance and cost of distributed data processing.
• Model data and maintain data warehouses for analytics.
• Collaborate with data teams and stakeholders in Agile teams.
• Ensure CI/CD and version control for data pipelines.

Conocimientos

Python

SQL

Spark

Databricks

Azure

ETL

Data pipelines

MLOps

CI/CD

Agile

Herramientas

Azure Synapse

Azure Data Factory

Azure Functions

MS SQL

ADLS Gen2

NoSQL

Descripción del empleo

The Hr Ally is seeking a Senior Azure Data Engineer / Lead in Sydney, Australia with 8+ years of software and data engineering experience. You will design, build and optimize large-scale data pipelines using PySpark, Databricks, Azure Synapse, and ADF, while collaborating with cross-functional teams in an Agile environment.

The role requires strong Python/SQL skills, data warehousing expertise, and familiarity with CI/CD and MLOps practices.