Senior Azure Data Engineer - ETL, PySpark & Data Platform

Umbrella Club; Sydney; On-site; AUD 140,000 - 200,000; Full time

Job summary

Umbrella Club in Sydney is seeking an experienced Senior Data Engineer to design, develop, and optimize modern data platforms and large-scale data solutions. You will build and maintain complex data pipelines and ETL processes, using Python, SQL, PySpark, Azure Synapse, ADF, and Azure Functions.

You will develop data models and data warehouses, work with MS SQL and ADLS Gen2, and manage distributed processing systems for batch workloads.

Qualifications

• 8+ years of software/data engineering experience.
• Hands-on Python and SQL; Java is a plus.
• Extensive PySpark, Azure Synapse, ADF, and Azure Functions experience.
• Strong ETL/data pipeline and data warehousing knowledge.
• Experience with MS SQL, ADLS Gen2, and NoSQL.
• Experience with distributed systems and batch processing.
• Knowledge of MLOps and Azure DevOps/Git.
• Strong problem solving and communication in Agile teams.

Responsibilities

• Design, develop, and maintain complex data pipelines and ETL processes.
• Build scalable data solutions using Python, SQL, PySpark, Azure Synapse, ADF, and Azure Functions.
• Develop and optimize data models, data warehouses, and large-scale processing.
• Work with SQL and NoSQL data stores including MS SQL and ADLS Gen2.
• Manage distributed data processing systems for batch processing.
• Implement data engineering aligned with MLOps and modern data architecture.
• Support data integration, transformation, and migration initiatives.
• Follow Agile development and collaborate with cross-functional teams.
• Use Azure DevOps and Git to support development and deployment.
• Contribute to data platform performance, scalability, and reliability.

Skills

Python

SQL

PySpark

Azure Synapse

ADF

Azure Functions

NoSQL

Data Modeling

ETL

MLOps

Azure DevOps

Git

Agile

Distributed Systems

Tools

MS SQL

ADLS Gen2

Job description

Umbrella Club in Sydney is seeking an experienced Senior Data Engineer to design, develop, and optimize modern data platforms and large-scale data solutions. You will build and maintain complex data pipelines and ETL processes, using Python, SQL, PySpark, Azure Synapse, ADF, and Azure Functions.

You will develop data models and data warehouses, work with MS SQL and ADLS Gen2, and manage distributed processing systems for batch workloads.