Fabric Data Engineer: Build Scalable Lakehouse Pipelines

Tata Consultancy Services; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Tata Consultancy Services (APAC) is hiring a Senior Data Engineer to design and operate end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory. You will develop Lakehouse architectures, ingest from multiple sources, and ensure data quality and governance across platforms.

Work involves implementing ETL/ELT with Spark and SQL, enabling Power BI reporting, and leading engineering teams. A strong background in Delta Lake, Purview, RBAC, and DevOps is required.

Qualifications

• Design enterprise data architecture leveraging Fabric components (Lakehouse, Warehouse, Dataflows Gen2, Pipelines, OneLake).
• Build and optimise Fabric Lakehouse tables, Warehouses, Pipelines, and Dataflows Gen2.
• Define data modelling standards, governance frameworks, and security patterns.
• Develop reusable ETL/ELT frameworks using PySpark, SQL, and Fabric notebooks.
• Manage CI/CD workflows via Azure DevOps and Git.
• Apply best practices for performance tuning, cost optimisation, and data quality.
• Provide technical leadership and guidance to engineering teams.
• Expertise in Delta Lake, Fabric Data Engineering, ADLS, PySpark/SQL.
• Strong understanding of data modelling and integration with Power BI datasets.
• Strong understanding of metadata management, Purview, access control, and FinOps.
• Certifications: DP-203, Fabric Analytics Engineer (preferred).

Responsibilities

• Design, build, and maintain end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory.
• Develop and manage Lakehouse and OneLake architectures for structured and unstructured data.
• Ingest data from multiple sources (databases, APIs, files, streaming sources) into Fabric.
• Implement ETL/ELT processes using Spark (PySpark) and SQL.
• Design and optimize data models to support analytics and reporting workloads.
• Develop and maintain Fabric Data Warehouse solutions using T-SQL.
• Ensure data quality, reliability, and consistency through validation and monitoring.
• Optimize pipeline performance, storage, and capacity utilization in Fabric.
• Implement security, access control (RBAC), and data sensitivity labels.
• Enable and support Power BI semantic models and downstream reporting use cases.
• Monitor, troubleshoot, and resolve data pipeline and performance issues.
• Implement data governance, lineage, and cataloging using Microsoft Purview.
• Support CI/CD and DevOps practices for analytics and data engineering workloads.
• Collaborate with architects, analysts, and stakeholders to translate business requirements into scalable data solutions.

Skills

Fabric architecture

Lakehouse

Data pipelines

PySpark

SQL

Power BI data

Delta Lake

Purview

RBAC security

CI/CD DevOps

Data modelling

Azure DevOps

Git

Tools

Azure Data Factory

Fabric notebooks

T-SQL

Git

DevOps

Job description

Tata Consultancy Services (APAC) is hiring a Senior Data Engineer to design and operate end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory. You will develop Lakehouse architectures, ingest from multiple sources, and ensure data quality and governance across platforms.

Work involves implementing ETL/ELT with Spark and SQL, enabling Power BI reporting, and leading engineering teams. A strong background in Delta Lake, Purview, RBAC, and DevOps is required.