Azure Data Engineer

Talworx Solutions; Kolkata District, Pune District, Delhi; Hybrid; INR 900,000 - 1,300,000; Full time

Job summary

Talworx Solutions in Kolkata is seeking a data engineer to design and implement large-scale data pipelines using Azure Data Factory, feeding data into Azure Data Lake Storage.

You will collaborate with cross-functional teams and optimize ETL processes with PySpark on Azure Databricks to handle massive datasets across nodes.

Qualifications

• Experience designing data pipelines in Azure-based environments.
• Strong ETL design and implementation capabilities.
• Hands-on experience with big data processing.

Responsibilities

• Design, develop, and maintain large-scale data pipelines using Azure Data Factory to ingest data into Azure Data Lake Storage.
• Collaborate with cross-functional teams to gather requirements and design data engineering solutions.
• Develop high-performance ETL processes using PySpark on Azure Databricks to process massive datasets in parallel.

Skills

Azure Data Factory

PySpark

Azure Databricks

ETL Design

Tools

Databricks

Azure Data Lake Storage

Job description

Roles and Responsibilities

• Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Data Lake Storage.
• Collaborate with cross-functional teams to gather requirements and design solutions for complex data engineering projects.
• Develop high-performance ETL processes using PySpark on Azure Databricks to process massive datasets in parallel across multiple nodes.