Senior AI Engineer: AWS Data Pipelines & RAG (Spark)

EPAM Systems Inc; United States; Remote; USD 150,000 - 210,000; Full time

Job summary

EPAM Systems is seeking a Senior AI Engineer to design and maintain data pipelines powering AI/GenAI applications for Risk-Based Quality Management in clinical trials.

Focus areas include RAG document ingestion, vector indexing, and AWS-based data infrastructure. The role requires strong Python, Spark SQL, and OpenSearch skills and collaboration with a cross-functional team in a fast-paced environment.

Qualifications

• 5+ years of hands-on data engineering experience at scale.
• Expertise in RAG document ingestion pipelines (chunking, embedding, vector indexing).
• Proficiency in AWS OpenSearch as a vector database for RAG workflows.
• Advanced proficiency in Python, including SQL and Spark SQL.
• Skills in unstructured data transformation (PDF, DOCX) for RAG/LLM applications.
• Familiarity with AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB.
• Knowledge of containerization with Docker.
• Proficiency in English at a B2+ level.
• Nice to have Background in pharmaceutical or life sciences domain.

Responsibilities

• Design and build RAG document ingestion pipelines (chunking, embedding, vector indexing) for clinical trial quality data.
• Build and manage vector databases (AWS OpenSearch) for RAG-powered AI workflows.
• Develop batch and streaming ETL/ELT pipelines from scratch for unstructured clinical data (PDF, DOCX, clinical reports).
• Build and expose data APIs for AI application consumption.
• Optimize chunking strategies, embedding generation, and retrieval performance for RAG architectures.
• Manage data quality, lineage, and governance for AI/ML data pipelines.
• Deploy and maintain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB).
• Collaborate with Data Scientists and Backend Developers in an integrated pod team.

Skills

RAG pipelines

AWS OpenSearch

Python

SQL

Spark SQL

Unstructured data

Docker

CI/CD

CDK

Terraform

CDISC standards

SageMaker

Snowflake

Pinecone

APIs

English (B2+)

Tools

Snowflake

Pinecone

SageMaker

OpenSearch

Docker

Terraform

CDK

Job description

EPAM Systems is seeking a Senior AI Engineer to design and maintain data pipelines powering AI/GenAI applications for Risk-Based Quality Management in clinical trials.

Focus areas include RAG document ingestion, vector indexing, and AWS-based data infrastructure. The role requires strong Python, Spark SQL, and OpenSearch skills and collaboration with a cross-functional team in a fast-paced environment.