Lead Databricks Engineer: Data Pipelines & ELT Architect

Capgemini; San Francisco (CA); Hybrid; USD 100,000 - 169,000; Full time

Benefits offered by this job

Comprehensive benefits

Hybrid work options

Job summary

Capgemini in Los Angeles, CA is seeking a Lead Data Engineer to design, build, and optimize scalable data ingestion and processing pipelines using Databricks. The role includes mentoring engineers and collaborating with business and technology stakeholders to deliver reliable, high-performing data solutions.

The position focuses on ELT/ETL design with Delta Lake, and integration from Oracle, MySQL, and SQL Server, across cloud-native platforms in a dynamic, inclusive environment.

Qualifications

• Experience designing and implementing enterprise-scale data ingestion frameworks.
• Proficient in batch and streaming data pipelines and cloud-native technologies.
• Strong leadership and mentoring abilities in a fast-paced environment.

Responsibilities

• Design and implement enterprise-scale data ingestion frameworks using Databricks Native Data Loader, Lakeflow Connect, and other Databricks-native capabilities.
• Build and optimize batch and streaming pipelines from Oracle, MySQL, SQL Server, APIs, and other sources.
• Develop robust ELT/ETL pipelines with Databricks, Spark, and Delta Lake.
• Implement end-to-end data movement, transformation, validation, and monitoring solutions.
• Lead architectural decisions related to data ingestion, storage, processing, and governance.
• Provide technical leadership and mentoring to junior engineers and collaborate with stakeholders.

Skills

Data engineering

ETL/ELT design

Leadership/mentoring

Tools

Databricks

Lakeflow Connect

Delta Lake

Job description

Capgemini in Los Angeles, CA is seeking a Lead Data Engineer to design, build, and optimize scalable data ingestion and processing pipelines using Databricks. The role includes mentoring engineers and collaborating with business and technology stakeholders to deliver reliable, high-performing data solutions.

The position focuses on ELT/ETL design with Delta Lake, and integration from Oracle, MySQL, and SQL Server, across cloud-native platforms in a dynamic, inclusive environment.