Lead AI Engineer: Spark & AWS RAG Pipelines

EPAM Systems, Inc.; Turkey; On-site; TRY 5,897,000 - 8,845,000; Full time

Job summary

EPAM Systems, Inc. is seeking a Lead AI Engineer to design and maintain data pipelines powering AI/GenAI apps for Risk-Based Quality Management in clinical trials.

You will lead RAG ingestion, vector indexing, and API development, while shaping data governance and AWS infrastructure (S3, Glue, Athena, Lambda, Step Functions, DynamoDB). Collaborate with Data Scientists and Backend Developers in a pod to deliver scalable, reliable data solutions for unstructured data (PDF, DOCX), using Spark SQL

Qualifications

• 7+ years of proven, large-scale data engineering experience.
• Expert in RAG document ingestion pipelines (chunking, embedding, vector indexing).
• Experience with AWS OpenSearch as vector database for RAG workflows.
• Strong Python, SQL, and Spark SQL skills.
• Experience transforming unstructured data (PDF, DOCX) for LLM/RAG apps.
• Knowledge of AWS services and containerization (Docker).

Responsibilities

• Architect and build RAG document ingestion pipelines (chunking, embedding, vector indexing).
• Set up vector databases (AWS OpenSearch) to support RAG workflows.
• Create batch and streaming ETL/ELT pipelines for unstructured clinical data.
• Construct and expose data APIs for AI applications.
• Improve chunking, embedding, and retrieval performance in RAG architectures.
• Oversee data quality, lineage, and governance across AI/ML data pipelines.
• Set up and sustain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB).
• Collaborate with Data Scientists and Backend Developers in a pod.

Skills

RAG pipelines

Python

SQL

Spark SQL

Data engineering

API development

Tools

AWS OpenSearch

Docker

Terraform

CDK

Jenkins

Bitbucket

SageMaker

Snowflake

Pinecone

Job description

EPAM Systems, Inc. is seeking a Lead AI Engineer to design and maintain data pipelines powering AI/GenAI apps for Risk-Based Quality Management in clinical trials.

You will lead RAG ingestion, vector indexing, and API development, while shaping data governance and AWS infrastructure (S3, Glue, Athena, Lambda, Step Functions, DynamoDB). Collaborate with Data Scientists and Backend Developers in a pod to deliver scalable, reliable data solutions for unstructured data (PDF, DOCX), using Spark SQL