Lead Data Engineer - Databricks & Data Ingestion

Capgemini; Northern, New York (KY, NY); Hybrid; USD 103,000 - 129,000; Full time

Benefits offered by this job

Paid time off

Medical, dental, and vision coverage

401(k)

Life and disability insurance

Employee assistance programs

Job summary

Capgemini is seeking a Lead Data Engineer in New York to design, build, and optimize scalable data ingestion pipelines using Databricks, Delta Lake, and cloud-native tools. You will lead data engineering efforts, mentor engineers, and collaborate with stakeholders to deliver reliable data solutions.

The role requires hands-on experience with Databricks Native Data Loader, Lakeflow Connect, Spark, and enterprise migrations from Oracle, MySQL, and SQL Server; strong leadership and CI/CD practices;

Qualifications

• Experience with Databricks Native Data Loader, Lakeflow Connect, and Delta Lake in enterprise environments.
• Hands-on data engineering with batch and streaming pipelines across heterogeneous sources.

Responsibilities

• Databricks Data Engineering & Data Ingestion: Design and implement enterprise-scale data ingestion frameworks using Databricks-native capabilities.
• Build and optimize batch and streaming data pipelines from Oracle, MySQL, SQL Server, and APIs.
• Develop robust ELT/ETL pipelines leveraging Databricks, Spark, Delta Lake, and cloud-native technologies.
• Lead architectural decisions related to data ingestion, storage, processing, and governance.
• Create and maintain scalable data pipeline architectures and ensure data quality.

Skills

Databricks

Spark

Delta Lake

ETL/ELT

Data ingestion

Oracle

MySQL

SQL Server

APIs

Databricks Native Data Loader

Tools

Lakeflow Connect

Job description

Capgemini is seeking a Lead Data Engineer in New York to design, build, and optimize scalable data ingestion pipelines using Databricks, Delta Lake, and cloud-native tools. You will lead data engineering efforts, mentor engineers, and collaborate with stakeholders to deliver reliable data solutions.

The role requires hands-on experience with Databricks Native Data Loader, Lakeflow Connect, Spark, and enterprise migrations from Oracle, MySQL, and SQL Server; strong leadership and CI/CD practices;