Senior AI Quality & Evaluation Engineer for RAG & Security

Solvd; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

Solvd Inc. is seeking a Senior AI Quality & Evaluation Engineer to own independent proof that retrieval works and that permissions hold. You’ll build the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates, ensuring engineers do not grade their own work.

This 50% allocation role spans weeks 3–10 and collaborates with the retrieval engineer and POD Lead. You’ll automate the QA harness, publish scores to the quality dashboard, and define

Qualifications

• 3+ years in software quality engineering or test automation with Python
• Hands-on evaluation of a search or RAG system in production
• Experience designing security-focused negative tests (authorization, entitlement, data exposure)
• Ability to work in AWS — Bedrock and OpenSearch APIs, CloudWatch, scheduled tasks

Responsibilities

• Build the golden question suite with sources, passages, and acceptance thresholds for relevance, grounding, and citation accuracy
• Create negative entitlement test set to ensure restricted passages are not returned
• Implement freshness and retirement tests to verify index lag and retirement behavior
• Automate the harness to run on schedule and after content/model changes, publish scores and trends
• Define QA gate content analysts apply to remediated content
• Run the evaluation suite before pilots in weeks 7–9 and report misses with root cause
• Write the evaluation runbook and train client stewards and platform owners

Skills

Python test automation

CI/CD

RAG/IR evaluation

Security negative tests

Technical reporting

Tools

AWS Bedrock

OpenSearch

CloudWatch

Job description

Solvd Inc. is seeking a Senior AI Quality & Evaluation Engineer to own independent proof that retrieval works and that permissions hold. You’ll build the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates, ensuring engineers do not grade their own work.

This 50% allocation role spans weeks 3–10 and collaborates with the retrieval engineer and POD Lead. You’ll automate the QA harness, publish scores to the quality dashboard, and define