Data Engineer

Protiviti India; Hyderabad, Pune District, Coimbatore District; Hybrid; INR 900,000 - 1,500,000; Full time

Job summary

Protiviti India is seeking a Data Engineer to design and operate a modern data platform on Microsoft Fabric, covering Lakehouse and Warehouse layers.

You will build robust ETL/ELT pipelines using PySpark, Fabric Notebooks, and Data Factory, while integrating CRM, ERP, and analytics datasets for self‑service BI and AI readiness.

Qualifications

• 4-5 years hands-on data engineering experience.
• Deep expertise with Microsoft Fabric including Lakehouse, Warehouse, Data Factory and Notebooks.
• Proficiency in PySpark and SQL for large-scale transformations.
• Experience designing Medallion (Bronze/Silver/Gold) architectures.
• Strong Python for scripting and orchestration.
• Experience with Azure cloud services (Data Lake Storage Gen2, Synapse, Data Factory, DevOps).
• Solid understanding of data warehousing concepts and data governance.

Responsibilities

• Design and maintain enterprise data warehouse on Microsoft Fabric; manage Lakehouse, Warehouse, Semantic layers.
• Build and operationalize ETL/ELT pipelines with PySpark, Fabric Notebooks, and Fabric Data Factory.
• Ensure performance tuning, query optimization, and capacity planning across Fabric environments.
• Create and maintain Power BI semantic models and dashboards from data engineering outputs.
• Document playbooks, blueprints, and standards to enable scalable, auditable infrastructure.

Skills

Fabric Lakehouse

PySpark

SQL

Medallion Architecture

Python

Azure

Data Warehousing

Power BI

Data Factory

Cloud

Education

Bachelor's or Master's in CS / Data Eng

Fabric Analytics Engineer / DP-203 certs

Tools

Microsoft Fabric

Azure Data Lake Storage

Azure Synapse

Fabric Notebooks

OneLake

Job description

JD: Data Engineer:

Responsibilities

Data Platform Design & Architecture

• Work with Engineering Lead and maintain the firm's enterprise data warehouse on Microsoft Fabric covering Lakehouse, Warehouse, and Semantic layers end-to-end.
• Maintain medallion-pattern data pipelines (Bronze Silver Gold) that consolidate data from practice verticals, internal operations, and client-facing systems.
• Define and enforce logical and physical schema design standards, ensuring data structures are optimized for both analytical queries and downstream AI consumption.
• Maintain architecture decision records, solution blueprints, and platform documentation to enable scalable, auditable infrastructure.

Pipeline Engineering & ETL/ELT Development

• Design, build, and operationalize robust ETL/ELT pipelines using PySpark, Fabric Notebooks, and Fabric Data Factory — handling ingestion, transformation, validation, and load.
• Integrate diverse data sources — CRM, ERP, project management tools, financial systems, and cloud-based SaaS applications — into a unified data platform.
• Implement incremental and batch ingestion patterns; design fault-tolerant pipelines with monitoring, alerting, and self-healing retry logic.
• Build and maintain reusable pipeline templates and modular transformation logic to accelerate future development.

Performance, Reliability & Platform Operations

• Own performance tuning, query optimization, indexing strategies, and capacity planning across the Fabric Lakehouse and Warehouse environments.
• Manage upgrade cycles, schema evolution, and version-controlled deployments using CI/CD best practices integrated with Azure DevOps.
• Implement SLA monitoring and data freshness alerting to ensure pipelines meet business-critical availability requirements.
• Conduct regular platform health reviews and proactively identify and resolve bottlenecks before they impact downstream consumers.

Analytics Enablement & Semantic Layer

• Build and maintain Power BI semantic models (datasets) on top of the Gold layer — ensuring business-friendly naming, consistent measures, and row-level security.
• Enable self-service analytics for consultants and operations teams by creating well-documented, reusable datasets that reduce dependency on the data team.
• Collaborate with analysts and practice leads to translate KPI definitions into reliable, calculated measures within the semantic layer.
• Develop and maintain interactive Power BI dashboards for practice performance, utilisation, revenue, and operational health.

Technology repository

• Document and maintain the firm's data engineering playbooks, onboarding guides, and standards so the platform can scale as the team grows.

Experience, Qualifications & Skills

Must-Have

• 4-5 years of hands-on experience in data engineering, data platform development, or related roles.
• Deep, practical expertise with Microsoft Fabric — Lakehouse, Warehouse, Data Factory, Fabric Notebooks, and OneLake.
• Proficiency in PySpark and SQL for large-scale data transformation, aggregation, and pipeline development.
• Proven experience designing and implementing Medallion (Bronze/Silver/Gold) architectures.
• Strong Python skills for scripting, automation, and data pipeline orchestration.
• Experience with Azure cloud services — Azure Data Lake Storage Gen2, Azure Synapse, Azure Data Factory, Azure DevOps.
• Solid understanding of data warehousing principles — schema design, ETL/ELT best practices, data quality, and lineage.
• Experience building Power BI semantic models and BI dashboards from data engineering outputs.

Good-to-Have

• Familiarity with Snowflake, Databricks, or other cloud-native data platforms.
• Exposure to data mesh architecture principles, data contracts, or federated governance models.
• Experience with streaming / real-time ingestion using Fabric Eventstream, Kafka, or Azure Event Hubs.
• Background in professional services, consulting, or internal analytics functions.

Key Competencies

MS Fabric (Lakehouse / Warehouse)

Azure Data / Synapse

PySpark / Spark

Data Factory / Pipelines

Power BI & Semantic Models

Medallion Architecture

ETL / ELT Design

Python & SQL

Data Governance & Lineage

Schema Design

Performance Tuning

KPI Frameworks

Educational Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, Statistics, or a related quantitative field.
• Relevant certifications valued: Microsoft Certified: Fabric Analytics Engineer Associate, Azure Data Engineer Associate (DP-203), or equivalent.
• A demonstrable portfolio of data platform builds, pipeline architectures, or open-source contributions is a compelling differentiator.