AI Data Engineer: Build Scalable ETL Pipelines for GenAI

Tata Consultancy Services Limited; Tampa (FL); On-site; USD 90,000 - 115,000; Full time

Job summary

Tata Consultancy Services Limited in Tampa, FL seeks an Engineer to design, develop and maintain scalable end-to-end data pipelines. You will build data processing solutions using Python, SQL and PySpark, integrating data from databases, APIs, and files for AI/ML and GenAI applications.

You will implement data-quality checks, optimize Spark workloads, and collaborate with AI/ML engineers and business stakeholders. A strong CI/CD/DevOps mindset and production support expertise are required.

Qualifications

• Bachelor's degree in Computer Science or equivalent.
• Strong hands-on experience with Python and SQL.
• Experience designing and developing ETL/ELT pipelines for large-scale data processing.
• Experience with AWS data services and Spark/PySpark.

Responsibilities

• Design, develop, and maintain scalable end-to-end ETL/ELT and data pipelines.
• Build data processing and transformation solutions using Python, SQL and PySpark.
• Develop batch and streaming pipelines to integrate data from databases, APIs, files and enterprise source systems.
• Implement data-quality checks, validation, reconciliation, deduplication and exception handling.
• Optimize data pipelines, SQL queries and Spark workloads for high-volume processing.
• Prepare and transform datasets required for AI/ML and Generative AI applications.
• Develop data ingestion and preparation pipelines supporting LLM/RAG solutions and vector databases.
• Collaborate with AI/ML engineers, application teams and business stakeholders on downstream data requirements.
• Implement automated deployment and monitoring through CI/CD and DevOps practices.
• Support production deployments, monitoring, troubleshooting and root-cause analysis for data-related issues.

Skills

Python

SQL

Data engineering

PySpark

Spark

Airflow

Data quality

Batch processing

CI/CD

Docker

Kubernetes

Education

Bachelor's degree in Computer Science

Tools

Apache Airflow

Kafka

Docker

Kubernetes

Job description

Tata Consultancy Services Limited in Tampa, FL seeks an Engineer to design, develop and maintain scalable end-to-end data pipelines. You will build data processing solutions using Python, SQL and PySpark, integrating data from databases, APIs, and files for AI/ML and GenAI applications.

You will implement data-quality checks, optimize Spark workloads, and collaborate with AI/ML engineers and business stakeholders. A strong CI/CD/DevOps mindset and production support expertise are required.