AI Tester-BSK 2nd stage-WFO-RAG test,chatbot-2nd Round F2F

V2soft; Mysuru, Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

V2Soft, Inc. is seeking an Evaluation Engineer in Mysuru to build robust ML evaluation pipelines, maintain coverage matrices, and run no-regression checks across model iterations. You will work with Python-based harnesses, CI/test automation, and QA reporting for customers.

The role requires hands-on experience with ML evaluation metrics, per-class analysis, and input on security testing and drift monitoring. On-site in Mysuru, with opportunities to collaborate across teams.

Qualifications

• Python, and the ability to write your own evaluation harnesses rather than only driving a test tool.
• Solid grasp of ML evaluation: precision, recall, per-class metrics, confidence intervals, sampling bias.
• Experience testing an ML or AI system in production — you have caught a model problem, not only an application bug.
• Test automation and CI, with pytest or equivalent.
• API testing and the discipline to build reproducible, versioned test datasets.
• Clear written reporting: your reports go to the customer and support a payment gate.

Responsibilities

• Build and maintain the coverage matrix across design types and symbol classes, flagging gaps.
• Report accuracy per symbol class with confidence intervals and enforce per-class minimums.
• Run no-regression checks across fine-tune rounds to prevent regression on rare classes.
• Stand up drift monitoring and periodic model-health reports.
• Run functional and integration testing of the HE2 workflow alongside model validation.
• Perform adversarial and AI security testing before production and after changes.
• Validate correction-quality controls and adjudicate conflicting corrections.

Skills

Python

ML evaluation

CI/Testing

pytest

Tools

SageMaker Model Monitor

MLflow

Job description

• Build and maintain the coverage matrix across design types and symbol classes, flagging thin or empty cells as known gaps
• Report accuracy per symbol class with confidence intervals, not just an aggregate figure, and enforce the per-class minimum instance count below which a class is reported as provisional
• Run no-regression checks across fine-tune rounds so a new model cannot pass on average while regressing on a rare but critical class
• Build the Knowledge Bot evaluation suite: a curated Q&A set with known-correct answers, scored for groundedness and faithfulness against every prompt and model version
• Perform adversarial and AI security testing before production and after material architectural change prompt injection, jailbreak attempts, RAG-specific attacks, cross-user data isolation and AI data leakage
• Validate correction-quality controls: spot-audit ambiguous classes, track inter-reviewer agreement, and confirm conflicting human corrections are adjudicated before entering the labelled set
• Stand up drift monitoring and produce the periodic model-health report covering drift indicators, correction rates, confidence-score trends, retraining events and acceptance results by version
• Run functional and integration testing of the HE2 workflow alongside the model validation work

What you need

• Python, and the ability to write your own evaluation harnesses rather than only driving a test tool
• Solid grasp of ML evaluation: precision, recall, per-class metrics, confidence intervals, sampling bias, and why a held-out set must stay held out
• Experience testing an ML or AI system in production — you have caught a model problem, not only an application bug
• Test automation and CI, with pytest or equivalent
• API testing and the discipline to build reproducible, versioned test datasets
• Clear written reporting: your reports go to the customer and support a payment gate

Nice to have

• LLM red-teaming or prompt injection testing experience
• Familiarity with RAG evaluation frameworks and groundedness scoring
• Statistical background: clustering effects, effective sample size, stratified sampling
• Exposure to SageMaker Model Monitor, MLflow or comparable drift and experiment tracking

Security testing certification or demonstrable practice