Data Engineer MS Fabrics

The HR Ally; Sydney; On-site; AUD 140,000 - 190,000; Full time

Job summary

The HR Ally is seeking a Senior Data Engineer in Sydney to design and maintain end-to-end data pipelines using Microsoft Fabric. You will build Lakehouse architectures, ingest data from diverse sources, and implement ETL/ELT with PySpark and SQL to support analytics and reporting workloads.

You will ensure data quality, governance, and secure access with Purview, RBAC, and data sensitivity labels, while collaborating with analysts and stakeholders.

Qualifications

• Experience designing enterprise data architectures with Fabric.
• Hands-on expertise with Delta Lake and Purview.
• Certifications: DP-203 or Fabric Analytics Engineer preferred.
• Strong SQL and PySpark skills.
• Ability to lead technical teams.

Responsibilities

• Design, build, and maintain end-to-end data pipelines using Fabric.
• Ingest data from multiple sources into Fabric.
• Implement ETL/ELT processes using PySpark and SQL.
• Develop Lakehouse and One Lake architectures for data assets.
• Ensure data quality, governance, and security using Purview and RBAC.
• Collaborate with stakeholders to translate requirements into scalable data solutions.
• Support CI/CD and DevOps practices for analytics workloads.

Skills

Data engineering

Fabric platforms

PySpark

SQL

CI/CD

Azure DevOps

Git

Delta Lake

Purview

Data modeling

Tools

Fabric Notebooks

Power BI

Azure Data Factory

Job description

• Design, build, and maintain end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory.
• Develop and manage Lakehouse and One Lake architectures for structured and unstructured data.
• Ingest data from multiple sources (databases, APIs, files, streaming sources) into Fabric.
• Implement ETL/ELT processes using Spark (Pyspark) and SQL.
• Design and optimize data models to support analytics and reporting workloads.
• Develop and maintain Fabric Data Warehouse solutions using T-SQL.
• Ensure data quality, reliability, and consistency through validation and monitoring.
• Optimize pipeline performance, storage, and capacity utilization in Fabric.
• Implement security, access control (RBAC), and data sensitivity labels.
• Enable and support Power BI semantic models and downstream reporting use cases.
• Monitor, troubleshoot, and resolve data pipeline and performance issues.
• Implement data governance, lineage, and cataloguing using Microsoft Purview.
• Support CI/CD and DevOps practices for analytics and data engineering workloads.
• Collaborate with architects, analysts, and stakeholders to translate business requirements into scalable data solutions.

Skill Experience requirement:

• Design enterprise data architecture leveraging Fabric components (Lakehouse, Warehouse, Dataflows Gen2, Pipelines, OneLake).
• Build and optimise Fabric Lakehouse tables, Warehouses, Pipelines, and Dataflows Gen2.
• Define data modelling standards, governance frameworks, and security patterns.
• Develop reusable ETL/ELT frameworks using PySpark, SQL, and Fabric notebooks.
• Manage CI/CD workflows via Azure DevOps and Git.
• Architect scalable, cost-efficient data solutions integrating Fabric with Azure/enterprise systems.
• Apply best practices for performance tuning, cost optimisation, and data quality.
• Provide technical leadership and guidance to engineering teams.
• Expertise in Delta Lake, Fabric Data Engineering, ADLS, PySpark/SQL.
• Strong understanding of data modelling and integration with Power BI datasets.
• Strong understanding of metadata management, Purview, access control, and FinOps.
• Certifications: DP-203, Fabric Analytics Engineer (preferred).