Data Engineer

KPI Partners; Pune District; On-site; INR 3,000,000 - 5,200,000; Full time

Job summary

KPI Partners in Pune, India is seeking an experienced Senior Microsoft Fabric Data Engineer to design, develop, and orchestrate scalable data pipelines that support enterprise analytics and reporting.

You will work with Oracle EBS and other enterprise data sources, implement data movement patterns, and collaborate with business stakeholders, data analysts, and engineering teams to deliver reliable, scalable data solutions.

Qualifications

• Hands-on experience with Microsoft Fabric.
• Experience building Data Factory pipelines in Microsoft Fabric.
• Experience with Notebook orchestration.
• Experience with Dataflow Gen2.
• Strong understanding of Copy Activity and data movement.
• Experience scheduling and orchestrating end-to-end data pipelines.
• Knowledge of Pipeline Parameters, Variables, and Dynamic Expressions.
• Experience implementing Retry Policies, Timeout Handling, and Failure Notifications.
• Experience integrating data from Oracle EBS.
• Excellent problem-solving and troubleshooting abilities.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Microsoft Fabric.
• Build and manage Data Factory pipelines within Microsoft Fabric.
• Schedule and orchestrate Notebooks, Copy Activities, and Dataflow Gen2 executions.
• Create and manage dependency chains between pipeline activities.
• Implement retry policies, timeout handling, and error recovery mechanisms.
• Configure dynamic pipelines using parameters, variables, and expressions.
• Set up automated notifications and alerts for pipeline failures.
• Design and orchestrate Bronze Silver Gold data processing workflows following the Medallion Architecture.
• Integrate and transform data from Oracle EBS and other enterprise data sources.
• Collaborate with business stakeholders, data analysts, and engineering teams to deliver reliable and scalable data solutions.
• Monitor, troubleshoot, and optimize data pipelines for performance and reliability.

Skills

Excellent problem-solving

Strong troubleshooting abilities

Tools

Microsoft Fabric

Data Factory Pipelines

Notebook orchestration

Dataflow Gen2

Copy Activity

Pipeline Parameters, Variables, and Dynamic Expressions

Retry Policies

Timeout Handling

Failure Notifications

Oracle EBS

Azure Data Services

PySpark

Spark

CI/CD for data engineering

Job description

We are looking for an experienced Senior Microsoft Fabric Data Engineer to join our growing Data Engineering team. The ideal candidate will have strong expertise in Microsoft Fabric, data pipeline development, and Oracle EBS integration. You will play a key role in designing, developing, and orchestrating scalable data pipelines to support enterprise analytics and reporting.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Microsoft Fabric.
• Build and manage Data Factory pipelines within Microsoft Fabric.
• Schedule and orchestrate Notebooks, Copy Activities, and Dataflow Gen2 executions.
• Create and manage dependency chains between pipeline activities.
• Implement retry policies, timeout handling, and error recovery mechanisms.
• Configure dynamic pipelines using parameters, variables, and expressions.
• Set up automated notifications and alerts for pipeline failures.
• Design and orchestrate Bronze Silver Gold data processing workflows following the Medallion Architecture.
• Integrate and transform data from Oracle EBS and other enterprise data sources.
• Collaborate with business stakeholders, data analysts, and engineering teams to deliver reliable and scalable data solutions.
• Monitor, troubleshoot, and optimize data pipelines for performance and reliability.

Mandatory Skills

• Strong hands-on experience with Microsoft Fabric
• Experience building Data Factory Pipelines in Microsoft Fabric
• Hands-on experience with Notebook orchestration
• Experience with Dataflow Gen2
• Strong understanding of Copy Activity and data movement
• Experience scheduling and orchestrating end-to-end data pipelines
• Knowledge of Pipeline Parameters, Variables, and Dynamic Expressions
• Experience implementing Retry Policies, Timeout Handling, and Failure Notifications
• Experience integrating data from Oracle EBS
• Excellent problem-solving and troubleshooting abilities

Good to Have

• Experience with Azure Data Services
• Knowledge of PySpark or Spark
• Experience with CI/CD for data engineering