Azure Databricks Engineer

Capgemini; Pune District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Capgemini in Pune is seeking a Data Engineer to design and develop end-to-end data pipelines using Azure Data Factory and Azure Databricks. The role involves working with Azure SQL options and Synapse analytics, and crafting data solutions with Python, PySpark, Scala, and Hive.

You will build scalable cloud data architectures, implement CI/CD for data apps, and collaborate with cross-functional teams while ensuring data quality and performance.

Responsibilities

• Design and develop end-to-end data pipelines using Azure Data Factory (ADF) and Azure Databricks (ADB).
• Work with Azure SQL Database, Azure SQL Data Warehouse, and Azure Synapse Analytics.
• Develop data processing solutions using Python, PySpark, Scala, and Hive.
• Build and maintain distributed data workflows on Azure Databricks.
• Implement CI/CD pipelines for data applications.
• Collaborate with teams to gather requirements and deliver high-quality data solutions.
• Ensure data quality, performance tuning, and cloud data architecture best practices.

Skills

Python

PySpark

Scala

Hive

Azure Data Factory

Azure Databricks

Azure SQL Database

Azure Synapse Analytics

CI/CD

Tools

CI/CD pipelines

Job description

Job Description

Your Role

• Design and develop end-to-end data pipelines using Azure Data Factory (ADF) and Azure Databricks (ADB).
• Work with Azure SQL Database, Azure SQL Data Warehouse, and Azure Synapse Analytics.
• Develop data processing solutions using Python, PySpark, Scala, and Hive.
• Build and maintain distributed data workflows on Azure Databricks.
• Implement CI/CD pipelines for data applications.
• Collaborate with teams to gather requirements and deliver high-quality data solutions.
• Ensure data quality, performance tuning, and cloud data architecture best practices.

Job Classification

Industry: IT Services & Consulting

Functional Area / Department: Engineering - Software & QA

Role Category: Software Development

Role: Data Engineer

Employement Type: Full time

Contact Details:

Company: Capgemini

Location(s): Pune

Ability to work in a team environment with members of varying skill levels. Highly motivated. Learns quickly.