Senior AI Engineer with Spark, AWS Services

EPAM Systems, Inc.; Turkey; On-site; TRY 6,866,000 - 9,809,000; Full time

Job summary

EPAM Systems, Inc. seeks a Senior AI Engineer to design and maintain data pipelines powering AI/GenAI in clinical trial risk management. The role emphasizes RAG ingestion, vector indexing, and building data APIs for AI apps within the RBQM Production Pod.

You will work with data scientists and backend engineers, optimizing retrieval, embeddings, and ETL/ELT pipelines using AWS services and Spark SQL.

Qualifications

• 5+ years of hands-on data engineering experience at scale.
• Expertise in RAG document ingestion pipelines (chunking, embedding, vector indexing).
• Proficiency in AWS OpenSearch as a vector database for RAG workflows.
• Advanced Python skills, including SQL and Spark SQL.
• Experience transforming unstructured data (PDF, DOCX) for LLM applications.
• Familiarity with AWS: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB.
• Containerization with Docker; building custom pipelines from scratch.

Responsibilities

• Design and build RAG document ingestion pipelines for clinical trial quality data.
• Build and manage vector databases for RAG-powered AI workflows.
• Develop batch and streaming ETL/ELT pipelines from unstructured data sources.
• Expose data APIs for AI application consumption.
• Optimize chunking, embeddings, and retrieval performance for RAG architectures.
• Ensure data quality, lineage, and governance for AI/ML pipelines.
• Collaborate with Data Scientists and Backend Developers in an integrated pod.

Skills

RAG ingestion pipelines

Python

Spark SQL

SQL

AWS OpenSearch

Docker

ETL/ELT

vector indexing

API development

SageMaker processing

Tools

AWS OpenSearch

S3

Lambda

Glue

Athena

DynamoDB

Bedrock

API Gateway

Git/Bitbucket

Terraform

CDK

Snowflake

Pinecone

Job description

We are seeking a Senior AI Engineer with Spark and AWS Services expertise to join the RBQM Production Pod within the program. You will build and maintain data pipelines that power AI/GenAI applications for Risk-Based Quality Management in clinical trials. This role focuses on RAG document ingestion, vector indexing, and building data APIs for AI applications.ResponsibilitiesDesign and build RAG document ingestion pipelines (chunking, embedding, vector indexing) for clinical trial quality dataBuild and manage vector databases (AWS OpenSearch) for RAG-powered AI workflowsDevelop batch and streaming ETL/ELT pipelines from scratch for unstructured clinical data (PDF, DOCX, clinical reports)Build and expose data APIs for AI application consumptionOptimize chunking strategies, embedding generation, and retrieval performance for RAG architecturesManage data quality, lineage, and governance for AI/ML data pipelinesDeploy and maintain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)Collaborate with Data Scientists and Backend Developers in an integrated pod teamRequirements5+ years of hands-on data engineering experience at scaleExpertise in RAG document ingestion pipelines (chunking, embedding, vector indexing)Proficiency in AWS OpenSearch as a vector database for RAG workflowsAdvanced proficiency in Python, including SQL and Spark SQLSkills in unstructured data transformation (PDF, DOCX) for RAG/LLM applicationsFamiliarity with AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDBKnowledge of containerization with DockerCapability to build custom pipelines from scratch, beyond configuring out-of-the-box servicesProficiency in English at a B2+ levelNice to haveBackground in pharmaceutical or life sciences domainFamiliarity with Snowflake, Pinecone (vector DB alternative)Knowledge of SageMaker processing jobsSkills in CI/CD tools (Jenkins, Git/Bitbucket) and infrastructure tools (CDK or Terraform)Understanding of clinical data standards (CDISC, ADaM, SDTM)We offerCONTINUOUS UPSKILLING, LEARNING & DEVELOPMENTDiversity of tasks and projectsAssessment center for objective review of competency levelPersonal development planMentoring programs and leadership developmentCertification and professional development supportAccess to learning platforms including more than 2,500 internal coursesEnglish courses taught by certified teachersCORPORATE BENEFITSExtra leave daysReferral bonusesCOMPENSATION PACKAGECompetitive compensation paid in USDRegular salary and performance reviewsMEDICAL & HEALTHCAREPrivate health insuranceWell-being eventsWORKING ENVIRONMENTRecreation areas and kitchensTea, coffee and snacksSports equipment and game consolesIT EquipmentMicrosoft’s Software Assurance Home Use Program (HUP)Please note that our Talent Attraction Team reviews applications and CVs submitted in English.EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.