Lead Databricks Data Engineer — Ingestion & Pipelines (Hybrid SF)

Capgemini; San Francisco (CA); Hybrid; USD 130,000 - 150,000; Full time

Benefits offered by this job

Paid time off

401(k) retirement plan

Health insurance

Job summary

Capgemini in San Francisco is seeking a Lead Data Engineer to design, build, and optimize scalable data ingestion and processing pipelines using Databricks native capabilities. You will lead architecture decisions, mentor engineers, and collaborate with stakeholders to deliver high‑performing data solutions.

Responsibilities include building batch and streaming pipelines from Oracle, MySQL, SQL Server and APIs, ensuring data quality, and driving DevOps and CI/CD practices across the data

Qualifications

• Experience designing large-scale data ingestion pipelines.
• Hands-on Databricks Spark Delta Lake knowledge.
• Proven leadership and mentoring skills.

Responsibilities

• Design and implement enterprise-scale data ingestion pipelines.
• Mentor engineers and lead data platform initiatives.
• Collaborate with stakeholders to deliver reliable data solutions.

Skills

Databricks

Data pipelines

ETL/ELT

Spark

Delta Lake

Oracle

MySQL

SQL Server

Cloud platforms

Tools

Lakeflow

Connect

Delta Lake

Job description

Capgemini in San Francisco is seeking a Lead Data Engineer to design, build, and optimize scalable data ingestion and processing pipelines using Databricks native capabilities. You will lead architecture decisions, mentor engineers, and collaborate with stakeholders to deliver high‑performing data solutions.

Responsibilities include building batch and streaming pipelines from Oracle, MySQL, SQL Server and APIs, ensuring data quality, and driving DevOps and CI/CD practices across the data