Senior Data Engineer (Pyspark, Microsoft Fabric)

Quest Global; Chennai District; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Quest Global in India is seeking a data engineering professional to design scalable data processing solutions using PySpark in Microsoft Fabric, and to optimize ETL/ELT pipelines for processing large-scale structured and semi-structured datasets with Spark DataFrames and Spark SQL.

You will develop data ingestion, transformation, and aggregation workflows using Fabric Notebooks, Lakehouse, and Data Factory Pipelines, ensuring reliable data delivery and high performance across enterprise data

Qualifications

• Hands-on experience in designing and developing scalable data processing solutions using PySpark in Microsoft Fabric.
• Experience building and optimizing ETL/ELT pipelines for processing large-scale structured and semi-structured datasets using Spark Data Frames and Spark SQL.
• Experience developing data ingestion, transformation, and aggregation workflows using Fabric Notebooks, Lakehouse, and Data Factory Pipelines.

Responsibilities

• Design and develop scalable data processing solutions with PySpark in Fabric.
• Build and optimize ETL/ELT pipelines for large-scale datasets.
• Develop ingestion, transformation, and aggregation workflows using Fabric Notebooks, Lakehouse, and Data Factory Pipelines.

Skills

PySpark

Microsoft Fabric

ETL/ELT pipelines

Spark DataFrames

Spark SQL

Job description

Role & responsibilities

• Hands-on experience in designing and developing scalable data processing solutions using PySpark in Microsoft Fabric.
• Experience building and optimizing ETL/ELT pipelines for processing large-scale structured and semi-structured datasets using Spark Data Frames and Spark SQL.
• Experience developing data ingestion, transformation, and aggregation workflows using Fabric Notebooks, Lakehouse, and Data Factory Pipelines.