Senior Databricks Data Engineer – Lakehouse Real-Time

Capgemini; New York (NY); On-site; USD 89,000 - 166,000; Full time

Benefits offered by this job

Paid time off

Health benefits

401(k) plan

Onsite work

Job summary

Capgemini is seeking a Senior Data Engineer to design, build, and maintain scalable batch and real-time data pipelines on the Databricks Lakehouse. You will implement ETL/ELT frameworks, ensure data quality, and support governance, security, and ML/AI operationalization.

The role emphasizes Medallion Architecture, Delta Lake, and advanced Databricks features, with onsite requirements in New York and Troy, MI. Collaboration with Data Science and AI teams is essential.

Qualifications

• 5+ years of experience in data engineering or related field.
• Experience designing and building scalable batch and real-time data pipelines.
• Strong fundamentals in data governance, security, and ML/AI operationalization.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
• Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
• Implement data quality checks, monitoring, and pipeline optimization for reliability and performance.
• Design and manage modern Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold).
• Develop and maintain Delta Lake tables with optimized query performance.
• Perform performance tuning with Databricks best practices (Photon Engine, Liquid Clustering, Partitioning).
• Configure and administer Unity Catalog for enterprise data governance.
• Implement security controls across catalog, schema, table, row-level, and column-level layers.
• Enable data lineage, auditing, and compliance across the data platform.
• Develop pipelines with Databricks Workflows and manage deployments using Databricks Asset Bundles (DABs).
• Implement CI/CD pipelines using GitHub, GitLab, and DevOps practices.
• Provision and manage infrastructure with Terraform and IaC methodologies.
• Prepare datasets for machine learning and generative AI applications; support feature engineering and feature stores.
• Enable vector search and retrieval for LLM and AI-driven use cases; collaborate with Data Science and AI teams.

Skills

PySpark

SQL

Databricks Lakehouse

Delta Lake

Unity Catalog

Databricks Workflows

Structured Streaming

Auto Loader

Medallion Architecture

Photon Engine

Liquid Clustering

Partitioning

Git

GitHub

GitLab

DevOps

CI/CD

Terraform

Infrastructure-as-Code

pytest

Tools

Databricks Lakehouse

Delta Lake

Unity Catalog

Databricks Workflows

GitHub

GitLab

DevOps

CI/CD

Terraform

Infrastructure-as-Code

pytest

Job description

Capgemini is seeking a Senior Data Engineer to design, build, and maintain scalable batch and real-time data pipelines on the Databricks Lakehouse. You will implement ETL/ELT frameworks, ensure data quality, and support governance, security, and ML/AI operationalization.

The role emphasizes Medallion Architecture, Delta Lake, and advanced Databricks features, with onsite requirements in New York and Troy, MI. Collaboration with Data Science and AI teams is essential.