Data Engineer- RAG

Space Inventive, Inc; Hyderabad; On-site; INR 1,800,000 - 2,400,000; Full time

Job summary

Space Inventive, Inc. in Hyderabad, India seeks a Senior Data Engineer to design, build, and maintain data pipelines powering AI/GenAI apps for RBQM in clinical trials. You will own RAG ingestion, vector indexing, and scalable data APIs in a modern AI data architecture stack.

The role emphasizes building custom pipelines from scratch, optimizing retrieval, and collaborating with Data Scientists in an integrated pod. Strong Python/SQL and AWS OpenSearch experience are required.

Qualifications

• 5-7+ years hands-on data engineering at scale.
• RAG document ingestion pipelines with vector indexing.
• AWS OpenSearch for vector databases in AI workflows.
• Python (advanced) including SQL/Spark SQL.
• Unstructured data transformation (PDF, DOCX) for RAG/LLM.
• AWS services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB.
• Containerization with Docker and building custom pipelines from scratch.

Responsibilities

• Design and build RAG document ingestion pipelines (chunking, embedding, vector indexing) for clinical trial quality data.
• Build and manage vector databases (AWS OpenSearch) for RAG-powered AI workflows.
• Develop batch and streaming ETL/ELT pipelines from scratch for unstructured clinical data (PDF, DOCX).
• Build and expose data APIs for AI application consumption.
• Optimize chunking strategies, embedding generation, and retrieval performance for RAG architectures.
• Manage data quality, lineage, and governance for AI/ML data pipelines.
• Deploy and maintain AWS data infrastructure (S3, Lambda, Glue, Athena, Bedrock, Step Functions, DynamoDB).
• Collaborate with Data Scientists and Backend Developers in an integrated pod team.

Skills

Python

SQL

Spark SQL

RAG architectures

Data pipelines

AI data architectures

Tools

AWS OpenSearch

S3

Glue

Athena

Bedrock

Step Functions

API Gateway

DynamoDB

Lambda

Containerization (Docker)

Job description

We are seeking a Senior Data Engineer to join the RBQM Production Pod. In this role, you will design, build, and maintain data pipelines that power AI/GenAI-driven applications for Risk-Based Quality Management (RBQM) in clinical trials. You will be responsible for RAG document ingestion, vector database indexing, and developing scalable data APIs to support AI applications. Strong expertise in modern AI data architectures is essential, as traditional ETL and data warehouse experience alone will not meet the requirements of this role.

Key Responsibilities

• Design and build RAG document ingestion pipelines (chunking, embedding, vector indexing) for clinical trial quality data
• Build and manage vector databases (AWS OpenSearch) for RAG-powered AI workflows
• Develop batch and streaming ETL/ELT pipelines from scratch for unstructured clinical data (PDF, DOCX, clinical reports)
• Build and expose data APIs for AI application consumption
• Optimize chunking strategies, embedding generation, and retrieval performance for RAG architectures
• Manage data quality, lineage, and governance for AI/ML data pipelines
• Deploy and maintain AWS data infrastructure (S3, Lambda, Glue, Athena, Step Functions, DynamoDB)
• Collaborate with Data Scientists and Backend Developers in an integrated pod team

Requirements

• 5-7+ years hands-on data engineering at scale
• RAG document ingestion pipelines (chunking, embedding, vector indexing) - traditional ETL/DWH experience alone is not sufficient
• AWS OpenSearch (vector database for RAG workflows)
• Python (advanced) including SQL/Spark SQL
• Unstructured data transformation (PDF, DOCX) for RAG/LLM applications
• AWS Services: S3, Lambda, Glue, Athena, Bedrock, Step Functions, API Gateway, CloudWatch, DynamoDB
• Containerization: Docker
• Must demonstrate building custom pipelines from scratch - not just configuring out-of-the-box services