Lead Data Engineer

EXL; Gurugram District, Bengaluru, Pune District; Hybrid; INR 800,000 - 1,400,000; Full time

Job summary

EXL is looking for a Data Engineer to develop and maintain complex ETL processes using PySpark on AWS Glue and Azure. You will ensure high-quality data processing through robust testing frameworks and collaborate with cross-functional teams to gather requirements for data modeling. Additionally, the role involves designing and managing large-scale data pipelines using Azure Data Factory. This position is based in Bengaluru, Karnataka, India.

Responsibilities

• Develop complex ETL processes using PySpark on AWS Glue/Azure.
• Ensure high-quality data processing by implementing robust testing frameworks.
• Collaborate with cross-functional teams to gather requirements.
• Design, develop, and maintain large-scale data pipelines using Azure Data Factory.

Job description

Roles and Responsibilities

• Develop complex ETL processes using PySpark on AWS Glue/Azure to integrate structured and unstructured data from multiple sources.
  
• Ensure high-quality data processing by implementing robust testing frameworks such as Databricks.
  
• Collaborate with cross-functional teams to gather requirements for data modeling and architecture design.
  
• Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure SQL DW.