Senior AI Quality & Evaluation Engineer (RAG QA)

Solvd, Inc.

United States

Remote

USD 120,000 - 180,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Solvd, Inc. is seeking a Senior AI Quality & Evaluation Engineer to own independent proof that retrieval works and that permissions hold.

You’ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work. This 50% allocation role spans weeks 3–10, requiring collaboration with retrieval engineers, content analysts, and client stewards.

Qualifications

  • 3+ years in software quality engineering or test automation with Python.
  • Experience evaluating a search or RAG system in production.
  • Experience designing security-focused negative tests (authorization, entitlement, data exposure).

Responsibilities

  • Build the golden question suite from top contact drivers and needs workshops — question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests.
  • Build the negative entitlement test set — queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm.
  • Build freshness and retirement tests — source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync.
  • Automate the harness to run on schedule and after every content, configuration, or model change — publishing scores to the quality dashboard with trend lines and regression alarms.
  • Define the QA gate content analysts apply to remediated content.
  • Run the suite before every pilot change in weeks 7–9 and report misses with root cause — content, chunking, ranking, or permissions — to the right owner.
  • Write the evaluation runbook and train client stewards and platform owners to add questions, re-run the suite, and interpret regressions.

Skills

Python
Test automation
CI/CD
Security testing
Technical reporting

Education

Bachelor's degree in Computer Science or related field

Tools

Python
OpenSearch
AWS Bedrock
CloudWatch

Job description

Solvd, Inc. is seeking a Senior AI Quality & Evaluation Engineer to own independent proof that retrieval works and that permissions hold.

You’ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work. This 50% allocation role spans weeks 3–10, requiring collaboration with retrieval engineers, content analysts, and client stewards.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Quality & Evaluation Engineer for RAG & Security
Senior AI Quality & Evaluation Engineer for RAG & Security

Solvd • United States

On-site
USD 120,000 - 180,000
Senior AI Quality & Evaluation Engineer
Senior AI Quality & Evaluation Engineer

Solvd • United States

On-site
USD 120,000 - 180,000
Senior RAG Architect & Retrieval AI Lead
Senior RAG Architect & Retrieval AI Lead

Salvo Software • United States

Remote
USD 180,000 - 240,000
Senior Product Software Engineer - AI Quality Engineering
Senior Product Software Engineer - AI Quality Engineering

Wolters Kluwer • Cape Girardeau (MO)

On-site
USD 120,000 - 150,000
Remote AI QA Architect & Evaluation Lead
Remote AI QA Architect & Evaluation Lead

Meridial • United States

Remote
USD 8,300 - 90,000
Senior AI SDET: Automate AI & RAG Testing
Senior AI SDET: Automate AI & RAG Testing

HighLevel • United States

Remote
USD 120,000 - 180,000
AI QA Engineer - Retrieval & Grounded Gen
AI QA Engineer - Retrieval & Grounded Gen

FDE Times • United States

Remote
USD 135,000 - 170,000
Career track opportunity
Health care for you and family
Paid maternity/paternity
+5
Senior AI QA Tester — Remote, LLM & RAG Validation
Senior AI QA Tester — Remote, LLM & RAG Validation

SIRO Clinpharm Pvt. Ltd. • United States

Remote
USD 90,000 - 130,000
Senior AI QA & Evaluation Engineer
Senior AI QA & Evaluation Engineer

Ruya • Maryland

Hybrid
USD 120,000 - 180,000
Health insurance
Dental coverage
Vision coverage
+3
Senior AI QA Engineer - AI Output Evaluation & Release
Senior AI QA Engineer - AI Output Evaluation & Release

Delta Exchange • United States

Remote
USD 120,000 - 160,000