Senior Data Engineer — Retrieval & RAG Platform

Solvd; United States; Remoto; USD 140.000 - 210.000; Tempo pieno

Descrizione del lavoro

solvd is seeking a software/ML engineer to own the retrieval workstream for an enterprise knowledge and agent-assistance solution. You will design and implement a versioned retrieval service with cited passages, ensuring permissions, latency, and cost targets are met.

You will build a hybrid vector/keyword search, deliver a Python API on Lambda behind API Gateway, and manage encryption, auth, and metrics. You’ll also create dashboards and runbooks for ongoing operations.

Competenze

• Four+ years of software or ML engineering experience.
• Experience with production RAG or enterprise search systems.
• Hands-on with AWS Bedrock Knowledge Bases and retrieval APIs.
• Proficient in Python and deploying Lambda APIs behind API Gateway.
• Infrastructure-as-code with Terraform or AWS CDK; CI for deployments.
• Understanding chunking, embeddings, reranking, latency and cost trade-offs.
• Experience building operational dashboards in CloudWatch or similar.

Mansioni

• Own retrieval workstream from cloud infra and design through API delivery and handover.
• Build hybrid vector and keyword search with metadata filters and ranking.
• Deliver versioned Python retrieval API using Lambda and API Gateway with auth and metrics.
• Apply permission metadata at query time to enforce access controls.
• Create reference retrieve-and-generate endpoint and support pilot testing.
• Develop runbooks and dashboards for model changes, re-embedding, index rebuilds and costs.

Conoscenze

Python

Lambda

API Gateway

Bedrock Knowledge Bases

OpenSearch Serverless

Embedding models

Terraform

AWS CDK

CI/CD

CloudWatch dashboards

Formazione

Bachelor’s degree or higher in CS/Math/related field

Strumenti

Terraform

AWS CDK

OpenSearch Serverless

Descrizione del lavoro

Role overview

Own the retrieval workstream for an enterprise knowledge and agent-assistance solution, from cloud infrastructure and search design through API delivery, evaluation, and operational handover. The core outcome is a versioned service that returns relevant, cited passages while enforcing caller permissions and meeting agreed latency and cost expectations.

Responsibilities

• Provision a managed knowledge base and vector search collection in the client’s AWS account using infrastructure code, with the specified embeddings, chunking approach, and metadata.
• Build hybrid vector and keyword search, apply metadata filters, and tune ranking against an evaluation question set.
• Deliver a versioned Python retrieval API using Lambda and API Gateway, including authentication, rate limits, usage metrics, structured errors, and citations with source details.
• Apply permission metadata at query time so callers only receive content they are authorized to access.
• Create a reference retrieve-and-generate endpoint and support pilot testing by investigating retrieval misses and refining results.
• Build operational quality dashboards and write runbooks for model changes, re-embedding, index rebuilds, costs, and service handover.

Requirements

• At least four years of software or machine-learning engineering experience, including a production RAG or enterprise search system evaluated with a repeatable test suite.
• Practical experience with Amazon Bedrock Knowledge Bases and retrieval APIs, embedding models, and OpenSearch Serverless vector search, including hybrid queries and metadata filters.
• Strong Python skills and experience deploying Lambda APIs behind API Gateway with authentication, versioning, and throttling.
• Infrastructure-as-code experience with Terraform or AWS CDK, plus CI for deployments into a client-owned AWS account.
• Understanding of chunking, embedding selection, reranking, and the trade-offs among relevance, latency, and cost.
• Experience creating operational dashboards in CloudWatch or a comparable tool.

Nice to have

• Relevant AWS certification in generative AI development or machine learning engineering.
• Familiarity with RAG evaluation measures such as faithfulness, context precision and recall, and citation accuracy.
• Experience providing APIs to contact-center agent-assistance products.
• Knowledge of Bedrock Guardrails and permission-aware retrieval for Microsoft 365 content.

Benefits and work setup

Work with an international team on applied AI projects, with an emphasis on ongoing learning, collaboration, and responsible AI practices.