AI Data Engineer - Databricks Pipelines (Onshore)

Capgemini; Dallas (TX); On-site; USD 54,000 - 85,000; Full time

Benefits offered by this job

Medical benefits

Retirement benefits

Career growth opportunities

Job summary

Capgemini is hiring a Store AI Agent - Onshore Data Engineer to design, build, and optimize scalable data and ML pipelines. You will leverage Databricks, PySpark, Python, and PyTorch to enable AI/ML initiatives, feature stores, and robust data platforms across cloud environments.

The role emphasizes data modeling, Delta Lake governance, and collaborative work with data scientists to operationalize models and ML workflows with strong DevOps practices.

Qualifications

• 5+ years of data engineering experience in scalable environments.
• 3+ years with Databricks and cloud-based data platforms.
• Experience supporting ML/AI use cases using PyTorch.
• Strong knowledge of data lakes, warehousing, and ETL/ELT patterns.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Python.
• Build batch and real-time data ingestion from diverse sources.
• Develop ETL/ELT solutions for analytics and AI/ML workloads.
• Implement data quality checks, monitoring, and observability.
• Optimize Databricks jobs, notebooks, and workflows for cost/performance.
• Design Delta Lake architectures and metadata management.
• Collaborate with data scientists to prepare datasets and feature stores.
• Support model training and inference infrastructure on Databricks.
• Integrate ML workflows with MLOps pipelines and CI/CD processes.
• Apply DataOps and MLOps best practices.

Skills

Python

PyTorch

PySpark

Spark

SQL

Data Modeling

REST APIs

Git

CI/CD

NLP

Education

Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or related field

Tools

Databricks Platform

Databricks Workflows

Delta Lake

Unity Catalog

AWS

Azure Data Factory

Docker

Kubernetes

Git

CI/CD

REST APIs

Job description

Capgemini is hiring a Store AI Agent - Onshore Data Engineer to design, build, and optimize scalable data and ML pipelines. You will leverage Databricks, PySpark, Python, and PyTorch to enable AI/ML initiatives, feature stores, and robust data platforms across cloud environments.

The role emphasizes data modeling, Delta Lake governance, and collaborative work with data scientists to operationalize models and ML workflows with strong DevOps practices.