AI Evaluation Engineer: Scale QA & Metrics

Appnovation Technologies; Phoenix (AZ); On-site; USD 100,000 - 160,000; Full time

Job summary

Appnovation Technologies is seeking a QA / AI Evaluation Engineer to lead large-scale evaluations, measure factual grounding, and build a robust metrics framework to track improvements in answer quality. You will work across ML systems and CI/CD pipelines to ensure reliable quality signals for product decisions.

The role requires strong Python skills, experience with evaluation frameworks, and collaboration with engineering teams to reproduce and verify fixes in a fast-paced environment.

Qualifications

• Bachelor’s Degree in a technical field or equivalent experience.
• 4+ years in QA / test engineering, with exposure to data/ML systems.
• Strong Python and data-science techniques for measuring factual grounding and answer quality.
• Experience with LLM evaluation frameworks and statistical analysis.
• Ability to build automated, large-scale eval harnesses plus lighter human-in-the-loop A/B tests.
• Comfort working across multiple LLM providers’ outputs.
• Test automation frameworks and scripting.
• Detail-oriented, with strong analytical and communication skills.

Responsibilities

• Run evals at scale across large question sets, from small human-UAT batches up to hundreds of thousands or millions of automated evaluations.
• Statistically measure factual grounding and accuracy lift (before/after), not just human A/B testing.
• Build a metrics framework showing quality improvement (e.g., “answer is X% supported by source content / Y% better”).
• Design load and quality tests as the corpus scales.
• Define and maintain test plans, test cases, and quality gates.
• Automate regression and evaluation suites; integrate them into CI/CD.
• Report quality metrics clearly to technical and non-technical stakeholders.
• Collaborate with engineering to reproduce, triage, and verify fixes.
• Continuously improve QA processes and coverage.

Skills

Python

Data science techniques

LLM evaluation frameworks

Statistical analysis

Test automation

Scripting

Communication

Cross-functional collaboration

QA / test engineering

Education

Bachelor’s Degree in a technical field

Tools

Python

CI/CD pipelines

Job description

Appnovation Technologies is seeking a QA / AI Evaluation Engineer to lead large-scale evaluations, measure factual grounding, and build a robust metrics framework to track improvements in answer quality. You will work across ML systems and CI/CD pipelines to ensure reliable quality signals for product decisions.

The role requires strong Python skills, experience with evaluation frameworks, and collaboration with engineering teams to reproduce and verify fixes in a fast-paced environment.