Hiring For Azure Data Engineer (Databricks & ADF)

WNS Holdings; Gurugram District; On-site; INR 1,400,000 - 2,000,000; Full time

Job summary

WNS Holdings is seeking a data engineer to design, build, and maintain scalable data pipelines using Azure Data Factory and Databricks. You will implement PySpark transformations, Spark SQL queries, and Databricks notebooks to process large datasets.

The role emphasizes data quality, transformation, and integration from multiple sources, with collaboration across analysts and developers and active involvement in CI/CD deployments.

Qualifications

• Bachelor’s or Master’s degree in Engineering or Technology required.
• 3–5 years of experience in data engineering or related field.
• Experience with Azure DP tools and Databricks is preferred.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Databricks.
• Develop data transformation logic using PySpark, Python, and Spark SQL.
• Build and maintain Databricks notebooks, workflows, and Spark-based data processing jobs.
• Develop and manage Azure Data Factory pipelines, including activities, parameters, triggers, and dependencies.
• Perform data cleansing, transformation, validation, and integration from multiple data sources.
• Optimize PySpark and Spark SQL jobs for performance and scalability.
• Troubleshoot pipeline failures and production issues with appropriate fixes.
• Collaborate with analysts, data engineers, developers, and stakeholders; document pipelines and processes.
• Support deployment and production releases across environments using CI/CD processes.

Skills

ETL/ELT pipelines

Azure Data Factory

Databricks

PySpark

Python

Spark SQL

CI/CD

Data quality

Education

Bachelor's or Master's degree in Engineering or Technology

Tools

Azure Data Factory

Databricks

Azure DevOps

Spark

Job description

Job Description

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Databricks.
• Develop data transformation and processing logic using PySpark, Python, and Spark SQL.
• Build and maintain Databricks notebooks, workflows, and Spark-based data processing jobs.
• Develop and manage Azure Data Factory pipelines, including activities, parameters, triggers, and dependencies.
• Perform data cleansing, transformation, validation, and integration from multiple data sources.
• Optimize PySpark and Spark SQL jobs for performance, scalability, and efficient resource utilization.
• Troubleshoot data pipeline failures and production issues and implement appropriate fixes.
• Work with large datasets and ensure data quality, accuracy, and reliability.
• Use Azure DevOps for source-code management, branching, pull requests, and CI/CD processes.
• Follow coding standards, best practices, and development processes.
• Collaborate with business analysts, data engineers, developers, and other stakeholders to understand requirements and deliver data solutions.
• Prepare technical documentation for pipelines, transformations, and data processes.
• Support deployment and production releases across different environments.

Qualifications

• Bachelor's or Master's degree in Engineering or Technology
• 3-5 years of experience