Senior Data Engineer - Databricks

Capgemini; New York (NY); On-site; USD 89,000 - 166,000; Full time

Benefits offered by this job

Paid time off

Health benefits

401(k) plan

Onsite work

Job summary

Capgemini is seeking a Senior Data Engineer to design, build, and maintain scalable batch and real-time data pipelines on the Databricks Lakehouse. You will implement ETL/ELT frameworks, ensure data quality, and support governance, security, and ML/AI operationalization.

The role emphasizes Medallion Architecture, Delta Lake, and advanced Databricks features, with onsite requirements in New York and Troy, MI. Collaboration with Data Science and AI teams is essential.

Qualifications

• 5+ years of experience in data engineering or related field.
• Experience designing and building scalable batch and real-time data pipelines.
• Strong fundamentals in data governance, security, and ML/AI operationalization.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
• Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
• Implement data quality checks, monitoring, and pipeline optimization for reliability and performance.
• Design and manage modern Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold).
• Develop and maintain Delta Lake tables with optimized query performance.
• Perform performance tuning with Databricks best practices (Photon Engine, Liquid Clustering, Partitioning).
• Configure and administer Unity Catalog for enterprise data governance.
• Implement security controls across catalog, schema, table, row-level, and column-level layers.
• Enable data lineage, auditing, and compliance across the data platform.
• Develop pipelines with Databricks Workflows and manage deployments using Databricks Asset Bundles (DABs).
• Implement CI/CD pipelines using GitHub, GitLab, and DevOps practices.
• Provision and manage infrastructure with Terraform and IaC methodologies.
• Prepare datasets for machine learning and generative AI applications; support feature engineering and feature stores.
• Enable vector search and retrieval for LLM and AI-driven use cases; collaborate with Data Science and AI teams.

Skills

PySpark

SQL

Databricks Lakehouse

Delta Lake

Unity Catalog

Databricks Workflows

Structured Streaming

Auto Loader

Medallion Architecture

Photon Engine

Liquid Clustering

Partitioning

Git

GitHub

GitLab

DevOps

CI/CD

Terraform

Infrastructure-as-Code

pytest

Tools

Databricks Lakehouse

Delta Lake

Unity Catalog

Databricks Workflows

GitHub

GitLab

DevOps

CI/CD

Terraform

Infrastructure-as-Code

pytest

Job description

This Senior Data Engineering role supports scalable batch and real-time data platforms on the Databricks lakehouse, with responsibilities spanning ETL/ELT development, governance and security, and support for machine learning and generative AI operationalization.

Location

• New York, NY (onsite)
• Troy, MI (onsite minimum four days per week)

Salary

USD 89,400 - 166,398 per year

Experience Level

5+ years of experience

Key Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
• Build robust ETL/ELT frameworks to ingest, transform, and process large-scale datasets from multiple sources.
• Implement data quality checks, monitoring, and pipeline optimization to improve reliability and performance.
• Design and manage modern Databricks Lakehouse solutions using Medallion Architecture (Bronze, Silver, Gold layers).
• Develop and maintain Delta Lake tables while ensuring optimal query performance.
• Perform performance tuning using Databricks best practices, including Photon Engine, Liquid Clustering, and Partitioning.
• Configure and administer Unity Catalog to manage data governance across the enterprise.
• Implement security controls including catalog, schema, table, row-level, and column-level security.
• Enable data lineage, auditing, and compliance requirements across the data platform.
• Develop and maintain pipelines using Databricks Workflows.
• Manage deployments using Databricks Asset Bundles (DABs).
• Implement CI/CD pipelines using GitHub, GitLab, and DevOps best practices.
• Provision and manage infrastructure using Terraform and Infrastructure-as-Code methodologies.
• Prepare and curate datasets for machine learning and generative AI applications.
• Support feature engineering and feature store implementations.
• Enable vector search and retrieval capabilities for LLM solutions and AI-driven use cases.
• Collaborate with Data Science and AI teams to operationalize ML workloads.

Required Technologies and Skills

• PySpark and SQL
• Databricks Lakehouse platform experience
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Structured Streaming
• Auto Loader
• Medallion Architecture (Bronze, Silver, Gold)
• Photon Engine, Liquid Clustering, and Partitioning
• Databricks Asset Bundles (DABs)
• Git, GitHub/GitLab, DevOps, and CI/CD
• Terraform and Infrastructure-as-Code
• pytest
• Cloud experience with at least one of: Azure, GCP, AWS
• LLM, vector search, and feature store

Benefits

• Paid time off based on employee grade (A-F), including Vacation (12-25 days), Company paid holidays, Personal Days, and Sick Leave
• Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs
• Other benefits as provided by local policy and eligibility

Onsite Work Requirement

Candidates must be able to work onsite at the customer location in Troy, MI a minimum of four (4) days per week.