Databricks Data Engineer - ML & Delta Lake Specialist

Capgemini; Dallas (TX); On-site; USD 112,611,000 - 175,936,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Retirement benefits

On-site Dallas, TX

Job summary

Capgemini in Dallas, TX seeks an Onshore Data Engineer focused on Databricks-based data and ML pipeline development. You design and optimize scalable data pipelines, ETL/ELT solutions, and Delta Lake architectures to support AI/ML workloads, feature stores, and model training.

The role collaborates with data scientists and ML engineers to prepare training datasets, optimize pipelines, and implement robust data governance, security, and observability practices on cloud platforms.

Qualifications

• Strong hands-on with the Databricks Platform.
• Expertise in Python and PySpark.
• Experience with PyTorch for ML workloads.
• Proficiency in Delta Lake, Unity Catalog, Databricks Workflows.
• Experience ETL/ELT pipelines, SQL, and data modeling.
• Experience with distributed computing and big data architectures.
• Knowledge of REST APIs and data integration patterns.
• Git and CI/CD tools experience.
• Experience with one or more cloud platforms (Azure, AWS, GCP).
• Experience with Spark, Kafka/Event Streaming, and Data Factory.
• Experience with Feature Stores and Data Lakes/Warehousing.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build batch and real-time data ingestion frameworks from diverse sources.
• Develop ETL/ELT solutions for warehousing, analytics, and AI/ML workloads.
• Implement data quality checks, monitoring, and observability solutions.
• Optimize Databricks jobs, notebooks, and workflows for performance and cost.
• Design Delta Lake architectures, data models, and metadata management.
• Collaborate with Data Scientists and ML Engineers on training datasets.
• Support deployment of deep learning solutions using PyTorch.
• Enable model training, experimentation, and inference infrastructure on Databricks.
• Integrate ML workflows with MLOps pipelines and CI/CD processes.
• Apply DataOps and MLOps best practices; automate deployments.
• Use cloud-native services for storage, security, monitoring, and orchestration.
• Translate business requirements into technical data solutions; ensure governance.

Skills

Databricks Platform

Python

PySpark

PyTorch

Delta Lake

Unity Catalog

Databricks Workflows

ETL/ELT

SQL

REST APIs

Git

CI/CD

Azure/AWS/GCP

Spark

Kafka/Event Streaming

Azure Data Factory

Data Lake

Data Warehousing

Feature Stores

5+ years

3+ years Databricks

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Docker

Kubernetes

Job description

Capgemini in Dallas, TX seeks an Onshore Data Engineer focused on Databricks-based data and ML pipeline development. You design and optimize scalable data pipelines, ETL/ELT solutions, and Delta Lake architectures to support AI/ML workloads, feature stores, and model training.

The role collaborates with data scientists and ML engineers to prepare training datasets, optimize pipelines, and implement robust data governance, security, and observability practices on cloud platforms.