Lead AI Engineer with Spark, AWS Services

EPAM Systems, Inc.; Turkey; On-site; TRY 5,897,000 - 8,845,000; Full time

Job summary

EPAM Systems, Inc. is seeking a Lead AI Engineer to design and maintain data pipelines powering AI/GenAI apps for Risk-Based Quality Management in clinical trials.

You will lead RAG ingestion, vector indexing, and API development, while shaping data governance and AWS infrastructure (S3, Glue, Athena, Lambda, Step Functions, DynamoDB). Collaborate with Data Scientists and Backend Developers in a pod to deliver scalable, reliable data solutions for unstructured data (PDF, DOCX), using Spark SQL

Qualifications

• 7+ years of proven, large-scale data engineering experience.
• Expert in RAG document ingestion pipelines (chunking, embedding, vector indexing).
• Experience with AWS OpenSearch as vector database for RAG workflows.
• Strong Python, SQL, and Spark SQL skills.
• Experience transforming unstructured data (PDF, DOCX) for LLM/RAG apps.
• Knowledge of AWS services and containerization (Docker).

Responsibilities

• Architect and build RAG document ingestion pipelines (chunking, embedding, vector indexing).
• Set up vector databases (AWS OpenSearch) to support RAG workflows.
• Create batch and streaming ETL/ELT pipelines for unstructured clinical data.
• Construct and expose data APIs for AI applications.
• Improve chunking, embedding, and retrieval performance in RAG architectures.
• Oversee data quality, lineage, and governance across AI/ML data pipelines.
• Set up and sustain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB).
• Collaborate with Data Scientists and Backend Developers in a pod.

Skills

RAG pipelines

Python

SQL

Spark SQL

Data engineering

API development

Tools

AWS OpenSearch

Docker

Terraform

CDK

Jenkins

Bitbucket

SageMaker

Snowflake

Pinecone

Job description

We're looking for a Lead AI Engineer skilled in Spark and AWS Services to become part of the RBQM Production Pod within the program. In this position, you'll construct and sustain data pipelines that drive AI/GenAI applications supporting Risk-Based Quality Management for clinical trials. The primary focus of this role includes RAG document ingestion, vector indexing, and developing data APIs for AI applications.ResponsibilitiesArchitect and construct RAG document ingestion pipelines (chunking, embedding, vector indexing) to support clinical trial quality dataEstablish and oversee vector databases (AWS OpenSearch) to support RAG-driven AI workflowsCreate batch and streaming ETL/ELT pipelines from the ground up for unstructured clinical data (PDF, DOCX, clinical reports)Construct and expose data APIs that AI applications can consumeEnhance chunking strategies, embedding generation, and retrieval performance within RAG architecturesOversee data quality, lineage, and governance across AI/ML data pipelinesSet up and sustain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)Partner with Data Scientists and Backend Developers as part of a unified pod teamRequirementsMinimum 7 years of practical, large-scale data engineering experienceStrong background in RAG document ingestion pipelines (chunking, embedding, vector indexing)Skilled in using AWS OpenSearch as a vector database for RAG workflowsHigh-level command of Python, along with SQL and Spark SQLExperience transforming unstructured data (PDF, DOCX) for use in RAG/LLM applicationsWorking knowledge of AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDBUnderstanding of Docker-based containerizationAbility to develop custom pipelines from the ground up, going beyond simple configuration of pre-built servicesProficiency in English at a B2+ levelNice to haveExperience within the pharmaceutical or life sciences sectorExposure to Snowflake and Pinecone (as an alternative vector database)Understanding of SageMaker processing jobsProficiency with CI/CD tools (Jenkins, Git/Bitbucket) and infrastructure-as-code tools (CDK or Terraform)Familiarity with clinical data standards (CDISC, ADaM, SDTM)We offerCONTINUOUS UPSKILLING, LEARNING & DEVELOPMENTDiversity of tasks and projectsAssessment center for objective review of competency levelPersonal development planMentoring programs and leadership developmentCertification and professional development supportAccess to learning platforms including more than 2,500 internal coursesEnglish courses taught by certified teachersCORPORATE BENEFITSExtra leave daysReferral bonusesCOMPENSATION PACKAGECompetitive compensation paid in USDRegular salary and performance reviewsMEDICAL & HEALTHCAREPrivate health insuranceWell-being eventsWORKING ENVIRONMENTRecreation areas and kitchensTea, coffee and snacksSports equipment and game consolesIT EquipmentMicrosoft’s Software Assurance Home Use Program (HUP)Please note that our Talent Attraction Team reviews applications and CVs submitted in English.EPAM is global leader in AI transformation engineering and integrated consulting, serving Forbes Global 2000 companies and ambitious startups. With over thirty years of expertise in custom software, product and platform engineering, we empower our clients to become AI-Native enterprises, driving measurable value from innovation and digital investments.