[SLTKGEUJ] AI Tester

ZettaMine Labs Pvt. Ltd.; Bengaluru; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

ZettaMine Labs Pvt. Ltd. in Bengaluru seeks an AI Evaluation & Harness Engineer to design and operate evaluation infrastructure and production harnesses for trustworthy AI agents.

You will set technical direction for AI quality, build evaluation datasets, and collaborate with engineers to close gaps between proven safety and ship-ready AI. The role demands hands-on experience with LLM systems in production, strong Python skills, and familiarity with evaluation frameworks.

Qualifications

• Bachelor's degree in Computer Science, Machine Learning, or related technical field, plus 3+ years of software engineering experience with 1+ years specifically building, evaluating, or productionizing LLM or agentic AI systems - or equivalent experience.
• Direct, hands-on experience integrating LLM APIs into production systems - this must be applied AI/ML engineering experience, not testing AI features from a pure QA standpoint.
• Practical experience with evaluation frameworks or techniques such as RAGAS, DeepEval, LLM-as-judge scoring, or comparable evaluation tooling.
• Strong Python engineering skills, including experience with structured-output/function-calling patterns and prompt versioning.

Responsibilities

• Design and build evaluation harnesses for LLM-powered workflows, including prompt-level regression testing and schema validation.
• Build and maintain LLM-as-judge evaluation pipelines with rubric design and inter-rater reliability checks.
• Own evaluation for agentic systems using ADK, LangChain/LangGraph, or comparable orchestration frameworks.
• Design and maintain CI/CD-style regression gates for prompt changes, model updates, and context changes.
• Manage model lifecycle, drift detection, and regression suites against gold datasets.

Skills

Python engineering

LLM integration

Evaluation frameworks

Education

Bachelor's degree in Computer Science/ML or related field

Tools

RAGAS

DeepEval

LangChain/LangGraph

Job description

Notice Period:Immediate Joiners to 25 Days NP Serving Candidates

Job Description

AI Evaluation & Harness Engineer - Healthcare AI

We are looking for an AI Evaluation & Harness Engineer to design and operate both halves of this system: the evaluation infrastructure that proves our AI is trustworthy, and the production harness - prompts, tool contracts, guardrails, context management - that makes our agents perform reliably in the first place. This is a hands-on technical role for someone who has actually built and evaluated LLM or agentic systems in production, not only tested them from the outside. You will set technical direction for how AI quality is measured and gated, build reusable evaluation datasets and judges, and work directly with the engineers building our agents to close gaps between what ships and what’s actually been proven safe.

Responsibilities

• Design and build evaluation harnesses for LLM-powered and agentic workflows, covering prompt-level regression testing, structured-output/schema validation, retrieval quality (context precision/recall), faithfulness, and answer relevancy, using frameworks such as RAGAS and DeepEval to operationalize these metrics rather than building scoring logic from scratch each time.
• Build and maintain LLM-as-judge evaluation pipelines with rubric design, calibration against human review, and inter-rater reliability checks, distinguishing dimensions that require deterministic assertions from those that genuinely require semantic judgment.
• Own evaluation and testing for agentic systems built on frameworks such as Google's Agent Development Kit (ADK), LangChain/LangGraph, or comparable orchestration frameworks - including trajectory evaluation, tool-call accuracy (tool selection vs. parameter construction, evaluated independently), agent handoff accuracy in multi-agent workflows, and failure-recovery testing via deliberate failure injection.
• Design and maintain the regression-gated evaluation pipeline that runs on every prompt change, model version bump, or context update - functioning as the AI equivalent of a CI/CD quality gate - and isolate model changes from prompt changes when both occur together rather than validating only at the end-to-end layer.
• Own model lifecycle and drift management: treat prompts as versioned behavioral specifications, run continuous (not just release-time) regression suites against golden datasets, and detect silent model drift introduced by upstream provider updates before it surfaces as a user-facing quality problem.
• Build and evolve the production harness around our agent Lambdas/services: structured-output contracts, prompt/context versioning, guardrails against unsafe or out-of-scope input, escalation logic for low-confidence outputs, and the observability/tracing layer that captures every input - prompt - model response - downstream action as an inspectable record.
• Build a structured failure taxonomy specific to our systems - grounding/hallucination failures, reasoning and instruction-drift failures, safety/adversarial failures, coordination failures in multi-agent handoffs, and stochastic/consistency failures - and use it to drive systematic test coverage rather than ad hoc test-writing.
• Design human-in-the-loop review workflows for low-confidence or high-stakes outputs, including sampling strategy, escalation paths, and feedback loops that route confirmed failures back into evaluation datasets.
• Convert every significant production incident into permanent evaluation coverage: identify why existing evaluations didn't catch it, close the gap, and confirm the fix before redeployment.
• Establish evaluation datasets, judges, rubrics, and regression baselines as shared, versioned infrastructure usable by other project teams, rather than infrastructure rebuilt per project.
• Produce audit-ready documentation of evaluation methodology, results, and drift trends to support compliance and clinical-safety review.

Qualifications

Required:

• Bachelor's degree in Computer Science, Machine Learning, or related technical field, plus 3+ years of software engineering experience with 1+ years specifically building, evaluating, or productionizing LLM or agentic AI systems - or equivalent experience.
• Direct, hands-on experience integrating LLM APIs (OpenAI, Anthropic, Gemini, or similar) into production systems - this must be applied AI/ML engineering experience, not testing AI features from a pure QA standpoint.
• Practical experience with evaluation frameworks or techniques such as RAGAS, DeepEval, LLM-as-judge scoring, or comparable evaluation tooling.
• Strong Python engineering skills, including experience with structured-output/function-calling patterns and prompt versioning.

Preferred:

• Experience with agent orchestration frameworks such as Google's Agent Development Kit (ADK), LangChain/LangGraph, or similar.
• Cloud platform experience - GCP (Vertex AI, Cloud Functions), AWS (Lambda, Bedrock), or Azure (AI Foundry) - is a strong plus, particularly for serverless AI workloads.
• Experience evaluating RAG pipelines or multi-agent systems in a regulated or safety-critical domain (healthcare, finance).
• Familiarity with HIPAA/PHI handling requirements and healthcare data governance.
• Experience designing failure taxonomies, trajectory/handoff test scenarios, and severity-tiered quality gates for agentic systems.
• Exposure to observability/tracing tooling for multi-step AI pipelines (e.g., LangSmith, Datadog LLM Observability, or equivalent).

Our AI Evaluation practice sits at the point where AI ambition meets AI accountability - every agent we ship, every model we adopt, and every prompt we change passes through the systems this role builds. If you've been in the room building and debugging LLM systems and want to make that discipline rigorous and repeatable at scale, this is that role