Store AI Agent - Onshore Data Engineer

Capgemini; Dallas (TX); On-site; USD 112,611,000 - 175,936,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Retirement benefits

On-site Dallas, TX

Job summary

Capgemini in Dallas, TX seeks an Onshore Data Engineer focused on Databricks-based data and ML pipeline development. You design and optimize scalable data pipelines, ETL/ELT solutions, and Delta Lake architectures to support AI/ML workloads, feature stores, and model training.

The role collaborates with data scientists and ML engineers to prepare training datasets, optimize pipelines, and implement robust data governance, security, and observability practices on cloud platforms.

Qualifications

• Strong hands-on with the Databricks Platform.
• Expertise in Python and PySpark.
• Experience with PyTorch for ML workloads.
• Proficiency in Delta Lake, Unity Catalog, Databricks Workflows.
• Experience ETL/ELT pipelines, SQL, and data modeling.
• Experience with distributed computing and big data architectures.
• Knowledge of REST APIs and data integration patterns.
• Git and CI/CD tools experience.
• Experience with one or more cloud platforms (Azure, AWS, GCP).
• Experience with Spark, Kafka/Event Streaming, and Data Factory.
• Experience with Feature Stores and Data Lakes/Warehousing.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python.
• Build batch and real-time data ingestion frameworks from diverse sources.
• Develop ETL/ELT solutions for warehousing, analytics, and AI/ML workloads.
• Implement data quality checks, monitoring, and observability solutions.
• Optimize Databricks jobs, notebooks, and workflows for performance and cost.
• Design Delta Lake architectures, data models, and metadata management.
• Collaborate with Data Scientists and ML Engineers on training datasets.
• Support deployment of deep learning solutions using PyTorch.
• Enable model training, experimentation, and inference infrastructure on Databricks.
• Integrate ML workflows with MLOps pipelines and CI/CD processes.
• Apply DataOps and MLOps best practices; automate deployments.
• Use cloud-native services for storage, security, monitoring, and orchestration.
• Translate business requirements into technical data solutions; ensure governance.

Skills

Databricks Platform

Python

PySpark

PyTorch

Delta Lake

Unity Catalog

Databricks Workflows

ETL/ELT

SQL

REST APIs

Git

CI/CD

Azure/AWS/GCP

Spark

Kafka/Event Streaming

Azure Data Factory

Data Lake

Data Warehousing

Feature Stores

5+ years

3+ years Databricks

Education

Bachelor’s or Master’s degree in Computer Science or related field

Tools

Docker

Kubernetes

Job description

On-site contract opportunity in Dallas, TX for an Onshore Data Engineer focused on Databricks-based data and ML pipeline development.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, and Python
• Build and optimize batch and real-time data ingestion frameworks from diverse data sources
• Develop ETL/ELT solutions for enterprise data warehousing, analytics, and AI/ML workloads
• Implement data quality checks, validation frameworks, monitoring, and observability solutions
• Optimize Databricks jobs, notebooks, clusters, and workflows for performance and cost efficiency
• Design and maintain Delta Lake architectures, data models, and metadata management solutions
• Collaborate with Data Scientists and ML Engineers to prepare training datasets and feature stores
• Support development and deployment of deep learning solutions using PyTorch
• Build scalable feature engineering pipelines and data transformation frameworks
• Enable model training, experimentation, and inference infrastructure on Databricks
• Integrate ML workflows with MLOps pipelines and CI/CD processes
• Apply DataOps and MLOps best practices
• Automate deployments using CI/CD pipelines
• Use cloud-native services for storage, security, monitoring, and orchestration
• Support infrastructure-as-code and environment automation initiatives
• Translate business, analytics, and data science requirements into technical data solutions
• Support compliance with data governance, privacy, and security standards
• Participate in architecture reviews and technology evaluations

Requirements

• Strong hands-on experience with the Databricks Platform
• Expertise in Python
• Strong hands-on experience with PySpark
• Experience with PyTorch for machine learning and deep learning workloads
• Knowledge of Delta Lake, Unity Catalog, and Databricks Workflows
• Experience building ETL/ELT pipelines
• Strong SQL skills and data modeling experience
• Experience with distributed computing and big data architectures
• Knowledge of REST APIs and data integration patterns
• Git version control and CI/CD tools experience
• Experience with one or more: Microsoft Azure (preferred), AWS, GCP
• Experience with one or more: Spark, Kafka/Event Streaming, Azure Data Factory or equivalent integration tools, Data Lake architectures, Data Warehousing solutions, Feature Stores
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field
• 5+ years of experience in Data Engineering
• 3+ years of experience with Databricks and cloud-based data platforms
• Experience supporting machine learning and AI use cases using PyTorch

Preferred Skills

• Experience with GenAI and LLM data pipelines
• Knowledge of MLflow and MLOps frameworks
• Hands-on experience with Docker and Kubernetes
• Experience with real-time streaming architectures
• Familiarity with vector databases and AI platforms
• Certifications in Databricks or cloud platforms

Nice-to-Have Experience

• Retail, QSR, Restaurant Technology, Digital Commerce, or Customer Analytics domains
• Large-scale enterprise data modernization initiatives
• AI/ML and advanced analytics implementations

Technologies

• Databricks Platform, Python, PySpark
• PyTorch
• Delta Lake, Unity Catalog, Databricks Workflows
• ETL/ELT, SQL
• REST APIs, Git, CI/CD
• Microsoft Azure, AWS, Google Cloud Platform (GCP)
• Spark, Kafka/Event Streaming, Azure Data Factory
• Feature Stores

Compensation and Benefits

• Pay range: $39.30/hour - $61.40/hour
• Benefits: medical, dental, vision, retirement benefits
• Job type: contract
• Location: Dallas, TX, United States (onsite)

Applications will be accepted on an ongoing basis.