Microsoft Fabric Data Engineer

Tata Consultancy Services; Sydney; On-site; AUD 120,000 - 180,000; Full time

Job summary

Tata Consultancy Services (APAC) is hiring a Senior Data Engineer to design and operate end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory. You will develop Lakehouse architectures, ingest from multiple sources, and ensure data quality and governance across platforms.

Work involves implementing ETL/ELT with Spark and SQL, enabling Power BI reporting, and leading engineering teams. A strong background in Delta Lake, Purview, RBAC, and DevOps is required.

Qualifications

• Design enterprise data architecture leveraging Fabric components (Lakehouse, Warehouse, Dataflows Gen2, Pipelines, OneLake).
• Build and optimise Fabric Lakehouse tables, Warehouses, Pipelines, and Dataflows Gen2.
• Define data modelling standards, governance frameworks, and security patterns.
• Develop reusable ETL/ELT frameworks using PySpark, SQL, and Fabric notebooks.
• Manage CI/CD workflows via Azure DevOps and Git.
• Apply best practices for performance tuning, cost optimisation, and data quality.
• Provide technical leadership and guidance to engineering teams.
• Expertise in Delta Lake, Fabric Data Engineering, ADLS, PySpark/SQL.
• Strong understanding of data modelling and integration with Power BI datasets.
• Strong understanding of metadata management, Purview, access control, and FinOps.
• Certifications: DP-203, Fabric Analytics Engineer (preferred).

Responsibilities

• Design, build, and maintain end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory.
• Develop and manage Lakehouse and OneLake architectures for structured and unstructured data.
• Ingest data from multiple sources (databases, APIs, files, streaming sources) into Fabric.
• Implement ETL/ELT processes using Spark (PySpark) and SQL.
• Design and optimize data models to support analytics and reporting workloads.
• Develop and maintain Fabric Data Warehouse solutions using T-SQL.
• Ensure data quality, reliability, and consistency through validation and monitoring.
• Optimize pipeline performance, storage, and capacity utilization in Fabric.
• Implement security, access control (RBAC), and data sensitivity labels.
• Enable and support Power BI semantic models and downstream reporting use cases.
• Monitor, troubleshoot, and resolve data pipeline and performance issues.
• Implement data governance, lineage, and cataloging using Microsoft Purview.
• Support CI/CD and DevOps practices for analytics and data engineering workloads.
• Collaborate with architects, analysts, and stakeholders to translate business requirements into scalable data solutions.

Skills

Fabric architecture

Lakehouse

Data pipelines

PySpark

SQL

Power BI data

Delta Lake

Purview

RBAC security

CI/CD DevOps

Data modelling

Azure DevOps

Git

Tools

Azure Data Factory

Fabric notebooks

T-SQL

Git

DevOps

Job description

Join Tata Consultancy Services, Asia Pacific and be part of an organization committed to sustainable development for our future. TCS follows the Tata group philosophy of building sustainable businesses that are rooted in the community and demonstrate care for the environment. Our unique values position us to combine a purpose-driven worldview with digital innovation, collaborating with customers, communities and governments to lead and shape innovative solutions for a sustainable future. TCS has been carbon neutral in its operations across 11 countries, 12 delivery centres and 18 offices in Asia Pacific since 2022. This is only the initial stage in TCS’ journey as we strive to achieve long-term net zero emissions by 2030.

Corporate sustainability is embedded in our triple-bottom-line, focusing on people, the planet, and our purpose. Our offices are designed with eco-friendly features that significantly reduce our carbon footprint and enhance energy efficiency. We actively champion green initiatives, such as promoting paperless operations, implementing energy-efficient practices, and fostering employee engagement in sustainability efforts. When you become part of the TCS family, you will play an essential role dedicated to innovation, excellence, and crafting a brighter, greener future together. Join us and be a part of our mission to drive sustainability through technology and talent at Tata Consultancy Services, APAC today.

Key Skills :

• Design enterprise data architecture leveraging Fabric components (Lakehouse, Warehouse, Dataflows Gen2, Pipelines, OneLake).
• Build and optimise Fabric Lakehouse tables, Warehouses, Pipelines, and Dataflows Gen2.
• Define data modelling standards, governance frameworks, and security patterns.
• Develop reusable ETL/ELT frameworks using PySpark, SQL, and Fabric notebooks.
• Manage CI/CD workflows via Azure DevOps and Git.
• Apply best practices for performance tuning, cost optimisation, and data quality.
• Provide technical leadership and guidance to engineering teams.
• Expertise in Delta Lake, Fabric Data Engineering, ADLS, PySpark/SQL.
• Strong understanding of data modelling and integration with Power BI datasets.
• Strong understanding of metadata management, Purview, access control, and FinOps.
• Certifications: DP-203, Fabric Analytics Engineer (preferred).

Key Responsibilities:

• Design, build, and maintain end-to-end data pipelines using Microsoft Fabric Data Engineering and Data Factory.
• Develop and manage Lakehouse and OneLake architectures for structured and unstructured data.
• Ingest data from multiple sources (databases, APIs, files, streaming sources) into Fabric.
• Implement ETL/ELT processes using Spark (PySpark) and SQL.
• Design and optimize data models to support analytics and reporting workloads.
• Develop and maintain Fabric Data Warehouse solutions using T-SQL.
• Ensure data quality, reliability, and consistency through validation and monitoring.
• Optimize pipeline performance, storage, and capacity utilization in Fabric.
• Implement security, access control (RBAC), and data sensitivity labels.
• Enable and support Power BI semantic models and downstream reporting use cases.
• Monitor, troubleshoot, and resolve data pipeline and performance issues.
• Implement data governance, lineage, and cataloging using Microsoft Purview.
• Support CI/CD and DevOps practices for analytics and data engineering workloads.
• Collaborate with architects, analysts, and stakeholders to translate business requirements into scalable data solutions.