Senior Data Engineer (GenAI / AWS)

BlackCube Labs; Los Angeles (CA); On-site; USD 140,000 - 195,000; Full time

Job summary

BlackCube Labs seeks a Senior Data Engineer to design and build scalable data pipelines and AI-powered document processing on AWS. The role focuses on end-to-end workflows that integrate generative AI, vector databases, and RAG architectures for intelligent agent systems.

The ideal candidate has 6–10 years of professional data engineering experience, strong Python/SQL/TypeScript skills, and hands-on AWS experience including Bedrock and Textract.

Qualifications

• 6–10 years of professional experience as a Data Engineer or similar role.
• Strong proficiency in Python, SQL, and TypeScript.
• Hands-on experience with AWS services: Lambda, S3, Step Functions, Glue, IAM, and Bedrock.
• Demonstrated expertise with vector databases and embedding technologies.
• Experience building RAG systems, knowledge bases, and semantic search solutions.
• Proficiency with AI/ML frameworks and libraries: LangChain, LlamaIndex, and Claude Agent SDK.
• Experience with Document AI, OCR technologies (Textract), and NLP services (Comprehend).
• Familiarity with infrastructure-as-code tools (Terraform, AWS CDK) and CI/CD automation (GitHub Actions).

Responsibilities

• Design and develop scalable data pipelines using AWS Step Functions, Lambda, S3, Textract, and Comprehend.
• Build automated document-processing workflows with chunking, embeddings, vector indexing, and semantic search.
• Develop and maintain vector databases and knowledge bases to support RAG systems and AI agents.
• Build and deploy AI agents using Claude Agent SDK and Amazon Bedrock AgentCore Runtime.
• Create and maintain CI/CD pipelines using GitHub Actions to automate builds, testing, and deployments.
• Optimize data pipelines for scalability, reliability, security, and performance across production environments.
• Implement infrastructure-as-code solutions using Terraform and AWS CDK for repeatable deployments.

Skills

Python

SQL

TypeScript

AWS Lambda

AWS S3

AWS Step Functions

AWS Glue

AWS IAM

Bedrock

Vector databases

Embeddings

RAG

LangChain

LlamaIndex

Claude Agent SDK

Terraform

AWS CDK

GitHub Actions

Textract

Comprehend

NLP

Tools

Pinecone

pgvector

OpenSearch

Job description

Our client is seeking a Senior Data Engineer with 6-10 years of experience to design and build advanced data pipelines and AI-powered document processing solutions on AWS. This role focuses on developing end-to-end workflows that integrate generative AI, vector databases, and RAG architectures to support intelligent agent systems.

Responsibilities & Qualifications

• Design and develop scalable data pipelines using AWS Step Functions, Lambda, S3, Textract, and Comprehend for extracting and processing structured and unstructured data
• Build automated document-processing workflows with document chunking, embeddings, vector indexing, and semantic search capabilities
• Develop and maintain vector databases and knowledge bases to support RAG (Retrieval-Augmented Generation) systems and integrate them with AI agents for contextual information retrieval
• Build and deploy AI agents using Claude Agent SDK and Amazon Bedrock AgentCore Runtime
• Create and maintain CI/CD pipelines using GitHub Actions to automate builds, testing, and deployments
• Optimize data pipelines for scalability, reliability, security, and performance across production environments
• Implement infrastructure-as-code solutions using Terraform and AWS CDK for repeatable, maintainable deployments

Requirements

• 6-10 years of professional experience as a Data Engineer or similar role
• Strong proficiency in Python, SQL, and TypeScript
• Hands-on experience with AWS services: Lambda, S3, Step Functions, Glue, IAM, and Bedrock
• Demonstrated expertise with vector databases (Pinecone, pgvector, OpenSearch) and embedding technologies
• Experience building RAG systems, knowledge bases, and semantic search solutions
• Proficiency with AI/ML frameworks and libraries: LangChain, LlamaIndex, and Claude Agent SDK
• Experience with Document AI, OCR technologies (Textract), and NLP services (Comprehend)
• Familiarity with infrastructure-as-code tools (Terraform, AWS CDK) and CI/CD automation (GitHub Actions)