AI QA Engineer

V2soft; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

V2soft in Bengaluru, Karnataka, is seeking a QA/ML evaluation engineer to own the acceptance testing for ML models. You will define accuracy metrics, enforce data splits, and build coverage matrices across design types and symbol classes.

You will report per-class performance with confidence intervals, run no-regression checks across fine-tune rounds, and contribute to security testing and drift monitoring before production releases.

Qualifications

• Strong Python skills and ability to write evaluation harnesses.
• Solid grasp of ML evaluation metrics and per-class analysis.
• Experience testing ML/AI systems in production environments.

Responsibilities

• Own the acceptance test plan and define detection accuracy goals.
• Build and maintain coverage matrices across design types and symbol classes.
• Report accuracy per class with confidence intervals and data counts.

Skills

Python

ML evaluation

CI & pytest

API testing

evaluation harnesses

Tools

SageMaker

MLflow

Job description

Work Description

• Own the acceptance test plan: define how 95% detection accuracy and 5% area variance are measured, and enforce the split between the calibration set used for tuning and the held-out set used for acceptance
• Build and maintain the coverage matrix across design types and symbol classes, flagging thin or empty cells as known gaps
• Report accuracy per symbol class with confidence intervals, not just an aggregate figure, and enforce the per-class minimum instance count below which a class is reported as provisional
• Run no-regression checks across fine-tune rounds so a new model cannot pass on average while regressing on a rare but critical class
• Build the Knowledge Bot evaluation suite: a curated Q&A set with known-correct answers, scored for groundedness and faithfulness against every prompt and model version
• Perform adversarial and AI security testing before production and after material architectural change prompt injection, jailbreak attempts, RAG-specific attacks, cross-user data isolation and AI data leakage
• Validate correction-quality controls: spot-audit ambiguous classes, track inter-reviewer agreement, and confirm conflicting human corrections are adjudicated before entering the labelled set
• Stand up drift monitoring and produce the periodic model-health report covering drift indicators, correction rates, confidence-score trends, retraining events and acceptance results by version
• Run functional and integration testing of the HE2 workflow alongside the model validation work

What you need

• Python, and the ability to write your own evaluation harnesses rather than only driving a test tool
• Solid grasp of ML evaluation: precision, recall, per-class metrics, confidence intervals, sampling bias, and why a held-out set must stay held out
• Experience testing an ML or AI system in production — you have caught a model problem, not only an application bug
• Test automation and CI, with pytest or equivalent
• API testing and the discipline to build reproducible, versioned test datasets
• Clear written reporting: your reports go to the customer and support a payment gate

Nice to have

• LLM red-teaming or prompt injection testing experience
• Familiarity with RAG evaluation frameworks and groundedness scoring
• Statistical background: clustering effects, effective sample size, stratified sampling
• Exposure to SageMaker Model Monitor, MLflow or comparable drift and experiment tracking
• Security testing certification or demonstrable practice