Senior Data Engineer - Climate Data Platform

Department of Climate Change, Energy, the Environment and Water; Parramatta; Hybrid; AUD 133.000 - 147.000; Vollzeit

Zusammenfassung

The NSW Department of Climate Change, Energy, the Environment and Water invites applications for a Senior Data Engineer in Parramatta. You will help shape the department’s modern data environment, design, develop, and support trusted data products used across climate, energy, and water management.

You will implement modern ingestion pipelines with Microsoft Fabric, build robust transformations in Fabric Notebooks, apply Medallion architecture, and ensure data quality, security, and governance

Qualifikationen

• Extensive experience in a data engineering role delivering production data solutions.
• Hands-on with Microsoft Fabric including Lakehouses, Warehouses, Data Pipeline, Mirroring, and Notebooks.
• Strong programming skills in Python, PySpark, and SQL.
• Experience developing scalable ETL/ELT processes with reusable, maintainable code.
• Knowledge of Delta Lake, schema evolution, Medallion architecture, and data quality practices.
• Experience implementing monitoring, auditing, error handling, and operational controls for data pipelines.
• Strong understanding of Microsoft Azure and Azure DevOps.
• Experience working in agile teams and collaborating with stakeholders.
• Ability translating business requirements into high-quality data products.

Aufgaben

• Design and deliver modern data ingestion pipelines using Microsoft Fabric Data Pipelines, Mirroring, and Eventstreams.
• Build robust transformations in Fabric Notebooks with modular, reusable PySpark and Python code.
• Apply Medallion architecture to create well-structured Lakehouses and Warehouses.
• Develop data products that are reliable, observable, and governed with strong data-quality controls.
• Monitor pipeline performance and resolve issues to protect downstream use.
• Collaborate with domain experts to translate data challenges into solutions aligned with data mesh principles.
• Share knowledge through notebooks, templates, frameworks to lift capability across the team.
• Support and operate the enterprise data platform including monitoring, access management, and incident resolution.

Kenntnisse

Python

PySpark

SQL

Agile teamwork

Data quality

Tools

Microsoft Fabric

Lakehouses

Warehouses

Data Pipeline

Mirroring

Notebooks

Microsoft Azure

Azure DevOps

Delta Lake

Jobbeschreibung

The NSW Department of Climate Change, Energy, the Environment and Water invites applications for a Senior Data Engineer in Parramatta. You will help shape the department’s modern data environment, design, develop, and support trusted data products used across climate, energy, and water management.

You will implement modern ingestion pipelines with Microsoft Fabric, build robust transformations in Fabric Notebooks, apply Medallion architecture, and ensure data quality, security, and governance