Forward Deployed Engineer - AI Assurance

Systems Limited; India; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Systems Limited is seeking a QA/testing engineer specializing in AI-native applications. You will design test plans, automation, and eval pipelines across model, prompt, and configuration changes.

You will work with AI architects to define testability, own the quality gate before production, and communicate risks to leadership while training delivery teams on AI-specific testing practices.

Qualifications

• 5–9 yrs QA/test engineering, with 2+ yrs AI/ML-powered feature testing.
• Proficient in Python-based test automation and CI/CD integration.
• Understands AI-specific failure modes — hallucination, bias, drift, non-determinism.
• Statistically literate to interpret model evaluation metrics beyond pass/fail.
• Familiar with red-teaming methodologies for AI systems.
• Clear, assertive communicator—willing to block a release over quality concerns.
• Detail-oriented and methodical under delivery pressure.
• Explains quality risk in business-impact terms, not just technical jargon.
• Hands-on evals engineering with frameworks such as OpenAI Evals, LangSmith, etc.
• Designs golden datasets and rubrics, calibrates LLM-as-judge scoring.

Responsibilities

• Build test plans and automation for AI-native features (functional + AI-specific).
• Run regression testing across model/prompt/config changes to catch drift.
• Red-team AI features for edge cases and adversarial inputs.
• Build automated eval pipelines integrated into CI/CD.
• Partner with AI Architects to define testability requirements before build starts.
• Own the quality gate before any AI feature ships to production.
• Communicate quality risk to delivery leadership in actionable terms.
• Train delivery teams on AI-specific testing practices.
• Own the evals framework with datasets, rubrics, and benchmarks per use case.
• Define eval thresholds per engagement and gate releases on them.
• Build eval tooling — dataset curation, trace capture, dashboards.
• Instrument production evals and drift monitoring, feeding failures back into datasets.

Skills

QA/testing

AI/ML testing

Test automation

Python

CI/CD

Red-teaming

Statistics

Communication

Eval frameworks

Golden datasets

Tools

OpenAI Evals

Ragas

DeepEval

LangSmith

Azure AI Foundry

Job description

Owns quality for AI-native applications — functional testing plus the AI-specific evaluation (accuracy, drift, hallucination).

KEY RESPONSIBILITIES

• Build test plans and automation for AI-native application features (functional + AI-specific)
• Run regression testing across model/prompt/config changes to catch silent quality drift
• Red-team AI features for edge cases and adversarial inputs where relevant
• Build automated eval pipelines integrated into CI/CD
• Partner with AI Architects to define testability requirements before build starts
• Own the quality gate before any AI feature ships to production
• Communicate quality risk to delivery leadership in terms they can act on
• Train delivery teams on AI-specific testing practices
• Own the evals framework for the practice — golden datasets, scoring rubrics, LLM-as-judge calibration, and versioned benchmarks per use case
• Define eval acceptance thresholds per engagement and gate releases on them
• Build eval engineering tooling — dataset curation, trace capture, offline/online eval runs, and dashboards delivery teams can read
• Instrument production evals and drift monitoring, feeding failures back into the golden datasets

REQUIREMENTS & SKILLS

• 5–9 yrs QA/test engineering, with 2+ yrs testing AI/ML-powered features specifically
• Strong test automation skills (Python-based frameworks, CI/CD integration)
• Understands AI-specific failure modes — hallucination, bias, drift, non-determinism — and designs tests for them
• Statistically literate enough to interpret model evaluation metrics, not just pass/fail results
• Familiarity with red-teaming methodologies for AI systems
• Clear, assertive communicator — willing to block a release over a quality concern
• Detail-oriented and methodical under delivery-timeline pressure
• Collaborative but independent — doesn't rubber-stamp under delivery pressure
• Explains quality risk in business-impact terms, not just technical jargon
• Hands-on evals engineering — builds and maintains eval suites with frameworks such as OpenAI Evals, Ragas, DeepEval, LangSmith, Azure AI Foundry evaluations
• Designs golden datasets and rubrics, and calibrates LLM-as-judge scoring against human review
• Understands RAG and agent eval metrics — groundedness, retrieval precision/recall, task completion, tool-call correctness, cost/latency
• Experience wiring evals and drift monitoring into CI/CD and production observability