AI QA Evaluation Engineer: Scale Metrics for ML

Appnovation Technologies; Greater London; On-site; GBP 60,000 - 90,000; Full time

Job summary

Appnovation Technologies is seeking a QA / AI Evaluation Engineer to join a forward-thinking data/ML focused team. You will run large-scale evals, measure factual grounding and accuracy lift, and build metrics frameworks to demonstrate quality improvements across AI outputs.

Applicants should have a strong background in QA/test engineering, Python, and experience with LLM evaluation frameworks, plus the ability to design automated evaluation harnesses and integrate them with CI/CD.

Qualifications

• 4+ years in QA / test engineering, with exposure to data/ML systems.
• Strong Python and data-science techniques for measuring factual grounding and answer quality.
• Experience with LLM evaluation frameworks and statistical analysis.

Responsibilities

• Run evals at scale across large question sets, from small batches to millions of evaluations.
• Statistically measure factual grounding and accuracy lift (before/after).
• Build a metrics framework showing quality improvements (X% supported by sources).
• Design load and quality tests as corpus scales.
• Define and maintain test plans, test cases, and quality gates.
• Automate regression and evaluation suites; integrate into CI/CD.
• Report quality metrics clearly to stakeholders.
• Collaborate with engineering to reproduce and verify fixes.
• Continuously improve QA processes and coverage.

Skills

QA testing

Python

LLM evaluation

Data/ML

Automation

Education

Bachelor’s Degree in a technical field

Tools

Python

CI/CD

LLM eval frameworks

Job description

Appnovation Technologies is seeking a QA / AI Evaluation Engineer to join a forward-thinking data/ML focused team. You will run large-scale evals, measure factual grounding and accuracy lift, and build metrics frameworks to demonstrate quality improvements across AI outputs.

Applicants should have a strong background in QA/test engineering, Python, and experience with LLM evaluation frameworks, plus the ability to design automated evaluation harnesses and integrate them with CI/CD.