Data Engineer (Microsoft Fabric)

Insight Advisors Private Limited; Pune District; On-site; INR 800,000 - 1,200,000; Part time

Job summary

Insight Advisors Private Limited is hiring a Data Engineer with experience in data engineering and cloud platforms. The role involves migrating SQL-based systems to Microsoft Fabric, focusing on ETL/ELT pipeline development and data quality assurance.

The ideal candidate should have 3–6 years of experience and expertise in SQL, PySpark, and cloud migrations. This is a contract position located in Pune, Maharashtra, and it offers a dynamic work environment and the chance to be part of significant data modernization projects.

Qualifications

• 3–6 years of experience in data engineering.
• Strong exposure to cloud data platforms.
• Experience with migration projects (AWS to Azure/Fabric preferred).

Responsibilities

• Convert SQL stored procedures and legacy transformation logic into PySpark / Spark SQL notebooks in Microsoft Fabric.
• Design and build end-to-end ETL/ELT pipelines using Fabric Data Factory and Dataflows Gen2.
• Develop Gold-layer data marts optimized for Power BI dashboards.

Skills

ETL/ELT pipeline development

Microsoft Fabric (Data Factory, Dataflows Gen2, Notebooks, OneLake)

PySpark

Spark SQL

SQL skills

Data modeling

Azure DevOps

Git

CI/CD pipelines

Education

B.E./B.Tech/MCA in Computer Science, IT, or Data Engineering

Job description

Job Description – Data Engineer (Microsoft Fabric – ETL & Migration)

Location

Pune Aundh

Experience

3–6 years of experience in data engineering, with strong exposure to cloud data platforms and migration projects (AWS to Azure/Fabric preferred).

Qualification

B.E./B.Tech/MCA/Relevant Degree in Computer Science, IT, Data Engineering, or related field

Contract

6 Month On Contract

Role Overview

We are looking for a Fabric Data Engineer to support enterprise data platform modernization by migrating legacy SQL-based systems to Microsoft Fabric. This role focuses on ETL/ELT pipeline development, converting SQL stored procedures into Fabric Spark notebooks, and delivering production‑grade data layers for analytics and reporting.

Key Responsibilities

• Convert SQL stored procedures and legacy transformation logic into PySpark / Spark SQL notebooks in Microsoft Fabric.
• Design and build end-to-end ETL/ELT pipelines using Fabric Data Factory, Dataflows Gen2, and Fabric pipelines.
• Develop Gold‑layer data marts optimized for Power BI dashboards and enterprise analytics.
• Implement data ingestion and transformation pipelines across Bronze, Silver, and Gold layers.
• Ensure data quality through validation, reconciliation, and comparison with legacy systems.
• Optimize data processing performance, query execution, and pipeline efficiency.
• Monitor and troubleshoot ETL pipelines, notebook runs, and data refresh failures.
• Implement CI/CD pipelines using Azure DevOps and Git for deployment across Dev, UAT, and Production environments.
• Collaborate with BI developers, business stakeholders, and ERP teams to translate business requirements into scalable data solutions.

Required Skills & Qualifications

• Strong experience in ETL/ELT pipeline development and data engineering.
• Hands‑on experience with Microsoft Fabric (Data Factory, Dataflows Gen2, Notebooks, OneLake).
• Strong expertise in PySpark and Spark SQL.
• Advanced SQL skills including stored procedures and query optimization.
• Strong understanding of data modeling (star schema, dimensional modeling).
• Experience with Azure DevOps, Git, and CI/CD pipelines.
• Understanding of medallion architecture (Bronze, Silver, Gold layers).

Key Success Metrics

• Successful conversion of SQL workloads into Fabric notebooks.
• Stable and scalable ETL pipelines with minimal failures.
• High‑quality, business‑ready datasets for reporting.
• Improved pipeline performance and reduced processing time.
• Reliable production data platform supporting enterprise analytics.