Azure Databricks Lead: Scalable Data Pipelines & DevOps

Capgemini; Jersey City (NJ); On-site; USD 103,330 - 128,656; Full time

Benefits offered by this job

Paid time off

Medical, dental, and vision coverage

Retirement savings plans

Job summary

Capgemini is seeking an experienced Azure Databricks Engineer in Jersey City, NJ. The role involves designing and optimizing ETL pipelines using Azure services, collaborating with Data Architects to enhance enterprise analytics.

The ideal candidate must possess solid expertise in PySpark, Delta Lake, and Azure Data Factory, ensuring effective data processing and governance while working in a dynamic environment.

Qualifications

• 5+ years of experience in data engineering or similar roles.
• Proven experience with cloud-based data solutions and platforms.
• Strong programming skills in PySpark and experience with Azure services.

Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Databricks.
• Build scalable data ingestion workflows from various sources.
• Implement transformation logic, data cleansing, enrichment, and validation.

Skills

Cloud-based data engineering

ETL development

Distributed data processing

PySpark

Delta Lake

Azure Data Factory

Data pipeline development

DevOps processes

Data governance

Education

Bachelor's degree in a relevant field

Tools

Azure Databricks

Azure Data Factory

Git

Azure DevOps

Job description

Capgemini is seeking an experienced Azure Databricks Engineer in Jersey City, NJ. The role involves designing and optimizing ETL pipelines using Azure services, collaborating with Data Architects to enhance enterprise analytics.

The ideal candidate must possess solid expertise in PySpark, Delta Lake, and Azure Data Factory, ensuring effective data processing and governance while working in a dynamic environment.