Microsoft Fabric - Senior Data Engineer

Tata Consultancy Services; Hyderabad, Chennai District, Bengaluru; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Tata Consultancy Services in Hyderabad, India seeks a Lead Data Engineer to design and build scalable data pipelines and enterprise data solutions using Microsoft Fabric. You will develop ETL/ELT processes, govern data with Fabric and Purview, and create semantic models and Power BI dashboards.

You will collaborate with data scientists, analysts and business stakeholders, architect Fabric workspaces, and oversee security, roles, and licenses while mentoring junior engineers.

Qualifications

• Experience designing scalable data solutions and architectures.
• Proficiency with Fabric and its data ecosystem.
• Knowledge of governance, security and privacy best practices.

Responsibilities

• Lead design and implementation of scalable data pipelines, warehouses, and enterprise data solutions using Fabric.
• Develop and optimize ETL/ELT processes to ingest, transform, and load data into Fabric.
• Establish data governance, security, and workspace architecture with access controls.
• Build semantic data models and dashboards using Fabric tools and Power BI.
• Collaborate with data scientists, analysts, and stakeholders to translate requirements.
• Architect and manage Fabric workspaces, roles, and licenses; monitor performance.
• Apply DevOps practices including CI/CD, version control, and automation using Python/PowerShell.
• Mentor junior engineers on Fabric technologies and best practices.

Skills

Data pipelines

ETL/ELT

Collaboration

Tools

Microsoft Fabric

Azure Data Factory

Synapse Analytics

OneLake

Power BI

Medallion Architecture

Microsoft Purview

Python

PowerShell

Terraform

Ansible

Job description

• Design & Architecture: Lead the design and implementation of scalable data pipelines, data warehouses, and enterprise-grade data solutions leveraging Microsoft Fabric.
• Data Pipeline Development: Develop and optimize complex ETL/ELT processes to ingest, transform, and load data from various sources into the Fabric environment.
• Data Governance & Security: Establish and enforce data integrity, quality, and security best practices, including workspace architecture, user access controls, and compliance with privacy regulations.
• Data Modeling & Reporting: Build and maintain semantic data models and interactive dashboards using Fabric tools and Power BI.
• Collaboration: Translate business requirements into technical solutions by working closely with data scientists, analysts, and business stakeholders.
• Platform Management: Architect and manage Fabric workspaces, user roles, permissions, and licenses to ensure efficient and secure operations.
• Monitoring & Optimization: Proactively monitor, troubleshoot, and fine-tune data pipelines and workflows for peak performance and efficiency.
• DevOps & Automation: Integrate DevOps practices, including CI/CD pipelines, version control, and automation using scripting languages like Python or PowerShell.
• Mentorship: Provide technical leadership and mentorship to junior engineers, transferring knowledge on Microsoft Fabric technologies and best practices.
• Key Technologies & Concepts
• Microsoft Fabric: The core platform for modern data analytics.
• Azure Data Factory & Synapse Analytics: Tools used for data integration and big data analytics within Fabric.
• OneLake: The unified data lake within Microsoft Fabric for storing and managing all types of data.
• Power BI: Used for building interactive dashboards and reports.
• Medallion Architecture: An architectural pattern for building clean and organized data lakes.
• Microsoft Purview: For data governance and compliance.
• Python & PowerShell: For automation and scripting.
• Terraform & Ansible: For Infrastructure as Code (IaC)