Senior AI Quality & Evaluation Engineer – RAG & Gatekeeper

Solvd, Inc.; United States; On-site; USD 110,000 - 190,000; Full time

Job summary

Solvd, Inc. is seeking a Senior AI Quality & Evaluation Engineer to own independent proofs that retrieval works and permissions hold. You’ll build the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work.

This is a 50% allocation role across weeks 3–10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day‑to‑day with the retrieval engineer.

Qualifications

• 3+ years in software quality engineering or test automation with Python.
• Hands-on evaluation of a search or RAG system in production — writing ground truth sets, scoring relevance and drift.
• Experience designing security-focused negative tests — authorization, entitlement, data exposure — and reporting them at the severity a client security team expects.
• Ability to work in AWS — invoking Bedrock and OpenSearch APIs, reading CloudWatch, deploying a scheduled job with infrastructure code.
• Clear written reporting; the suite’s output is read by client stewards, not only engineers.

Responsibilities

• Build the golden question suite from top contact drivers and needs workshops — question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests.
• Build the negative entitlement test set — queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm.
• Build freshness and retirement tests — source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync.
• Automate the harness to run on schedule and after every content, configuration, or model change — publishing scores to the quality dashboard with trend lines and regression alarms.
• Define the QA gate content analysts apply to remediated content.
• Run the suite before every pilot change in weeks 7–9 and report misses with root cause — content, chunking, ranking, or permissions — to the right owner.
• Write the evaluation runbook and train client stewards and platform owners to add questions, re‑run the suite, and interpret regressions.

Skills

Python

RAG systems evaluation

Security negative tests

AWS work

Clear reporting

Tools

Python

OpenSearch

Bedrock

CloudWatch

CI/CD

Job description

Solvd, Inc. is seeking a Senior AI Quality & Evaluation Engineer to own independent proofs that retrieval works and permissions hold. You’ll build the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work.

This is a 50% allocation role across weeks 3–10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day‑to‑day with the retrieval engineer.