Senior Azure Data Engineer

Umbrella Club; Sydney; Presencial; AUD 140.000 - 200.000; Jornada completa

Descripción de la vacante

Umbrella Club in Sydney is seeking an experienced Senior Data Engineer to design, develop, and optimize modern data platforms and large-scale data solutions. You will build and maintain complex data pipelines and ETL processes, using Python, SQL, PySpark, Azure Synapse, ADF, and Azure Functions.

You will develop data models and data warehouses, work with MS SQL and ADLS Gen2, and manage distributed processing systems for batch workloads.

Formación

• 8+ years of software/data engineering experience.
• Hands-on Python and SQL; Java is a plus.
• Extensive PySpark, Azure Synapse, ADF, and Azure Functions experience.
• Strong ETL/data pipeline and data warehousing knowledge.
• Experience with MS SQL, ADLS Gen2, and NoSQL.
• Experience with distributed systems and batch processing.
• Knowledge of MLOps and Azure DevOps/Git.
• Strong problem solving and communication in Agile teams.

Responsabilidades

• Design, develop, and maintain complex data pipelines and ETL processes.
• Build scalable data solutions using Python, SQL, PySpark, Azure Synapse, ADF, and Azure Functions.
• Develop and optimize data models, data warehouses, and large-scale processing.
• Work with SQL and NoSQL data stores including MS SQL and ADLS Gen2.
• Manage distributed data processing systems for batch processing.
• Implement data engineering aligned with MLOps and modern data architecture.
• Support data integration, transformation, and migration initiatives.
• Follow Agile development and collaborate with cross-functional teams.
• Use Azure DevOps and Git to support development and deployment.
• Contribute to data platform performance, scalability, and reliability.

Conocimientos

Python

SQL

PySpark

Azure Synapse

ADF

Azure Functions

NoSQL

Data Modeling

ETL

MLOps

Azure DevOps

Git

Agile

Distributed Systems

Herramientas

MS SQL

ADLS Gen2

Descripción del empleo

We are looking for an experienced Senior Data Engineer to join our team and contribute to the design, development, and optimization of modern data platforms and large-scale data solutions.

Key Responsibilities

• Design, develop, and maintain complex data pipelines and ETL processes.
• Build scalable data solutions using Python, SQL, Apache PySpark, Azure Synapse, ADF, and Azure Functions.
• Develop and optimize data models, data warehouses, and large-scale data processing solutions.
• Work with SQL and NoSQL data stores, including MS SQL and ADLS Gen2.
• Develop and manage distributed data processing systems and clusters for batch processing.
• Implement data engineering solutions aligned with MLOps and modern data architecture practices.
• Support data integration, transformation, and migration initiatives.
• Follow Agile development practices and collaborate with cross-functional teams.
• Use modern DevOps tools such as Azure DevOps and Git to support development and deployment.
• Contribute to data platform performance, scalability, and reliability improvements.

About You

• 8+ years of experience in software engineering/data engineering.
• Strong hands‑on experience with Python and SQL; Java experience is desirable.
• Extensive experience with Apache PySpark, Azure Synapse, Azure Data Factory (ADF), and Azure Functions.
• Strong experience building complex ETL/data pipelines and working with modern data modelling concepts.
• Experience with MS SQL, ADLS Gen2, NoSQL, and large‑scale data warehousing.
• Experience working with distributed systems and batch‑processing environments.
• Knowledge of MLOps architecture and practices.
• Good understanding of Azure DevOps and Git.
• Strong understanding of ETL and data warehousing methodologies.
• Strong problem‑solving and communication skills with experience working in Agile environments.

Location: Sydney