Senior Data Engineer

Pearson Carter; Wellington; On-site; NZD 150,000 - 210,000; Full time

Job summary

Pearson Carter is seeking an experienced Senior Data Engineer / Engineering Lead with deep expertise in the Microsoft Fabric ecosystem to deliver governed, production-ready data products for a critical public sector environment in Wellington, NZ.

You will lead end-to-end ingestion and ETL patterns, enforce Medallion Architecture standards, and ensure seamless integration with Microsoft Purview governance, OneLake, and Lakehouse components using PySpark, T-SQL, Python, and Delta Lake

Qualifications

• Experience with Microsoft Fabric platform (OneLake, Lakehouse, Dataflows Gen2, Pipelines, DirectLake/Semantic Models).
• Proficiency in PySpark, T-SQL, Python, and Delta Lake optimization.
• Experience with data governance and Purview integration.

Responsibilities

• Design, build, and deploy 1–2 production-grade data products within an MS Fabric platform.
• Implement Medallion Architecture across OneLake, Lakehouses, and Data Warehouses.
• Build scalable orchestration pipelines using MS Fabric Dataflows Gen2, Pipelines, PySpark/Notebooks, and Matillion.
• Collaborate on lineage tracking, security controls, and metadata capture with Purview.
• Participate in early-phase platform reviews to validate capacity and readiness.

Skills

MS Fabric Core

PySpark

Python

T-SQL

Delta Lake

Tools

Matillion

Microsoft Purview

Job description

We are seeking an experiencedSenior Data Engineer / Engineering Leadwith deep expertise in theMicrosoft Fabric ecosystemto deliver governed, production-ready data products for a critical enterprise public sector environment.

You will lead the implementation of end-to-end ingestion and ETL patterns, enforce Medallion Architecture standards (Bronze/Silver/Gold), and ensure seamless integration with Purview data governance frameworks.

Key Responsibilities

• Pipeline & Data Product Delivery:Design, build, and deploy 1-2 production-grade data products within an existing MS Fabric platform to validate core ingestion and ETL/ELT frameworks.
• Architecture Implementation:Implement Medallion Architecture patterns across OneLake, Lakehouses, and Data Warehouses to transform raw inputs into structured, business-ready datasets.
• Data Integration:Build and maintain scalable orchestration pipelines using MS Fabric Dataflows Gen2, Pipelines, PySpark/Notebooks, and integration tools such as Matillion.
• Governance & Metadata Support:Collaborate with the Data Architect and Metadata Lead to implement lineage tracking, security controls, and metadata capture natively integrated with Microsoft Purview.
• Environment Validation:Participate in early-phase platform reviews to validate capacity setup, workspace configuration, and technical readiness before scaling out.

Required Experience & Skills

• MS Fabric Core:Hands-on experience building solutions on Microsoft Fabric (OneLake, Lakehouse, Dataflows Gen2, Fabric Pipelines, DirectLake/Semantic Models).
• Data Engineering Stack:Strong proficiency in PySpark, T-SQL, Python, and Delta Lake optimizations (partitioning, OPTIMIZE/VACUUM patterns).
• ETL/ELT Tools:Experience with modern ingestion platforms, metadata-driven pipelines, and cloud connectors (experience with Matillion is a strong bonus).
• Data Architecture Knowledge:Practical application of Medallion Architecture standards and star-schema dimensional modeling.
• Governance Awareness:Familiarity with Microsoft Purview for data cataloging, sensitivity labeling, and access control execution.
• Public Sector / Enterprise background:Experience operating within complex enterprise frameworks, structured change processes, and data privacy environments.

Candidate Deliverables for Submission

Please provide profiles highlighting:

1. Specific MS Fabric / Azure Lakehouse projects executed.
2. Experience leading ingestion design (specifically ETL patterns and Medallion implementation).
3. Availability for an immediate part-time start leading into full-time work.