Sr. Data Engineer (GenAI & AWS) Remote/Hybrid with Travel

Simple Solutions; United States; Hybrid; USD 180,000 - 240,000; Full time

Job summary

BlackCube Labs is seeking a Senior Data Engineer (GenAI & AWS) to design and implement data pipelines across AWS services, focusing on Retrieval-Augmented Generation and AI agent integrations. The role requires extensive experience with vector databases, embeddings, and building scalable data architectures.

The candidate will work in a remote/hybrid setup with travel to client sites for workshops and meetings, prioritizing timezone coverage across the West Coast and beyond.

Qualifications

• 12+ years of experience in data engineering or data platform roles.
• Strong AWS and Python expertise with a focus on GenAI workflows.
• Experience designing data pipelines, ETL/ELT, and automation.
• Proficiency with vector databases, embeddings, and RAG patterns.
• Hands-on experience deploying AI agents and large language models.
• Familiarity with CI/CD pipelines and IaC tooling.
• Excellent collaboration and communication across time zones.

Responsibilities

• Design and develop data pipelines using AWS Step Functions, Lambda, S3, Textract, and Comprehend.

Skills

Python

SQL

Time zone coverage

Cloud computing

Communication

Python proficiency

Python scripting

Team collaboration

Tools

AWS Step Functions

AWS Lambda

AWS S3

Amazon Textract

Amazon Comprehend

Vector databases

Embeddings

RAG

Amazon Bedrock

Claude Agent SDK

AgentCore Runtime

GitHub Actions

CI/CD

AWS IAM

API integrations

TypeScript

AWS Glue

OpenSearch

Pinecone

pgvector

LangChain

LlamaIndex

Terraform

AWS CDK

Document AI / OCR

Claude Agent SDK

Job description

Sr. Data Engineer (GenAI & AWS) Remote/Hybrid with Travel

The key is time zone coverage and so we can expand our search across the West Coast - remote/hybrid but necessarily located in the western time zones. You must be comfortable to travel to the client site for workshops/meetings.

Senior Data Engineer (GenAI / AWS)

Overall experience: 12-20+ years Cloud Platform: AWS Primary Language: Python

Key Responsibilities

• Design and develop data pipelines using AWS Step Functions, Lambda, S3, Textract, and Comprehend.
• Build automated document-processing workflows for extracting, transforming, and processing structured and unstructured data.
• Implement document chunking, embeddings, vector indexing, and semantic search.
• Develop and maintain vector databases and knowledge bases to support Retrieval-Augmented Generation (RAG).
• Integrate knowledge bases with AI agents for contextual information retrieval.
• Build AI agents using Claude Agent SDK and deploy them on Amazon Bedrock AgentCore Runtime.
• Develop CI/CD pipelines using GitHub Actions to automate builds, testing, and deployments.
• Optimize data pipelines for scalability, reliability, security, and performance.

Must-have skills

• Python, SQL
• AWS Step Functions, Lambda, S3
• Amazon Textract and Comprehend
• Vector databases and embeddings
• Document chunking and indexing
• RAG and knowledge-base integration
• Amazon Bedrock
• GitHub Actions CI/CD
• AWS IAM and API integrations

Good-to-have skills

• TypeScript
• AWS Glue
• Document AI / OCR
• OpenSearch, Pinecone, pgvector
• Advanced retrieval and reranking
• LangChain / LlamaIndex
• Claude Agent SDK
• Terraform / AWS CDK
• AgentCore Runtime

Important: If this individual must personally build and deploy the Claude agents, move Claude Agent SDK and AgentCore Runtime into the must-have category.