Senior Databricks Engineer - Spark, Delta Lake & Cloud

Capgemini; Cary (NC); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Healthcare coverage

Paid time off

Parental leave

401(k) plan

Employee Resource Groups

Job summary

Capgemini is seeking a Databricks Engineer with 8+ years of experience to design and optimize scalable data solutions on the Databricks Lakehouse Platform. You will implement Delta Lake, Medallion Architecture, and end-to-end data pipelines, collaborating with architects and analysts.

The role requires strong Python/SQL skills, cloud experience (Azure/AWS/GCP), and a track record in enterprise ETL/ELT. Onsite/Hybrid from Day 1 in Cary, NC with full benefits.

Qualifications

• Bachelor's degree in CS/IT/engineering or related field.
• 8+ years of IT experience with data engineering and big data.
• 5+ years hands-on Databricks experience.
• Strong Spark, PySpark, Spark SQL, and Delta Lake skills.
• Proficiency in Python and SQL.
• Experience in enterprise-scale ETL/ELT solutions.
• Understanding of data warehousing, modeling, and lakehouse architecture.
• Hands-on cloud experience (Azure, AWS, or GCP).
• Experience with Git and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines on Databricks.
• Build and optimize ETL/ELT workflows for batch and streaming data.
• Develop data transformation using PySpark and Spark SQL.
• Implement Delta Lake and Medallion Architecture for reliable data platforms.
• Integrate data from databases, APIs, cloud storage, and streams.
• Collaborate with data architects and analysts to deliver data-driven solutions.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Implement data quality, governance, security, and monitoring frameworks.
• Support CI/CD automation, release management, and production deployments.
• Troubleshoot production issues and provide platform support.
• Document technical designs and operational procedures.

Skills

Databricks

PySpark

Spark SQL

Python

SQL

ETL/ELT Development

Data Warehousing

Data Modeling

Git

CI/CD

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or related field

Tools

Databricks

Git

Terraform

Airflow

Kafka

Job description

Capgemini is seeking a Databricks Engineer with 8+ years of experience to design and optimize scalable data solutions on the Databricks Lakehouse Platform. You will implement Delta Lake, Medallion Architecture, and end-to-end data pipelines, collaborating with architects and analysts.

The role requires strong Python/SQL skills, cloud experience (Azure/AWS/GCP), and a track record in enterprise ETL/ELT. Onsite/Hybrid from Day 1 in Cary, NC with full benefits.