AI Evaluation & Harness Engineer

Zettamine Labs; Bengaluru, Mumbai; Híbrido; INR 4.200.000 - 6.600.000; Jornada completa

Descripción de la vacante

Zettamine Labs is seeking an AI Evaluation & Harness Engineer to design and operate evaluation infrastructure for trustworthy AI and production harness prompts, tool contracts, and guardrails. You will help define how AI quality is measured, build evaluation datasets, and work with engineers to ensure safe, reliable agent performance.

Role requires hands-on production experience with LLMs/agentic systems, experience integrating LLM APIs, and familiarity with evaluation tooling.

Formación

• Bachelor's degree in CS/ML with 3+ years software engineering experience, and 1+ year building or productionizing LLM/agentic AI systems.
• Hands-on experience integrating LLM APIs into production systems, not just QA.
• Experience with evaluation frameworks such as RAGAS or DeepEval.

Responsabilidades

• Design and build evaluation harnesses for LLM-powered workflows and agentic systems.
• Develop LLM-as-judge evaluation pipelines with rubric design and calibration.
• Own evaluation for agentic systems using ADK, LangChain, or similar frameworks.
• Design the regression-gated evaluation pipeline that runs on prompts and model changes.
• Establish model lifecycle and drift management with versioned prompts and datasets.

Conocimientos

Python

LLM systems

API integration

Prompt versioning

Educación

Bachelor's degree in Computer Science, Machine Learning, or related technical field

Herramientas

RAGAS

DeepEval

ADK

LangChain

Descripción del empleo

Role

AI Evaluation & Harness Engineer Healthcare AI

Experience

7 to 11 Years

Location

Bangalore / Mumbai

Role Overview

We are looking for an AI Evaluation & Harness Engineer to design and operate both halves of this system: the evaluation infrastructure that proves our AI is trustworthy, and the production harness prompts, tool contracts, guardrails, context management that makes our agents perform reliably in the first place. This is a hands-on technical role for someone who has actually built and evaluated LLM or agentic systems in production, not only tested them from the outside. You will set technical direction for how AI quality is measured and gated, build reusable evaluation datasets and judges, and work directly with the engineers building our agents to close gaps between what ships and what's actually been proven safe.

Responsibilities

• Design and build evaluation harnesses for LLM-powered and agentic workflows, covering prompt-level regression testing, structured-output/schema validation, retrieval quality (context precision/recall), faithfulness, and answer relevancy, using frameworks such as RAGAS and DeepEval to operationalize these metrics rather than building scoring logic from scratch each time.
• Build and maintain LLM-as-judge evaluation pipelines with rubric design, calibration against human review, and inter-rater reliability checks, distinguishing dimensions that require deterministic assertions from those that genuinely require semantic judgment.
• Own evaluation and testing for agentic systems built on frameworks such as Google's Agent Development Kit (ADK), LangChain/LangGraph, or comparable orchestration frameworks including trajectory evaluation, tool-call accuracy (tool selection vs. parameter construction, evaluated independently), agent handoff accuracy in multi-agent workflows, and failure-recovery testing via deliberate failure injection.
• Design and maintain the regression-gated evaluation pipeline that runs on every prompt change, model version bump, or context update functioning as the AI equivalent of a CI/CD quality gate and isolate model changes from prompt changes when both occur together rather than validating only at the end-to-end layer.
• Own model lifecycle and drift management: treat prompts as versioned behavioral specifications, run continuous (not just release-time) regression suites against golden datasets, and detect silent model drift introduced by upstream provider updates before it surfaces as a user-facing quality problem.
• Build and evolve the production harness around our agent Lambdas/services: structured-output contracts, prompt/context versioning, guardrails against unsafe or out-of-scope input, escalation logic for low-confidence outputs, and the observability/tracing layer that captures every input prompt model response downstream action as an inspectable record.
• Build a structured failure taxonomy specific to our systems grounding/hallucination failures, reasoning and instruction-drift failures, safety/adversarial failures, coordination failures in multi-agent handoffs, and stochastic/consistency failures and use it to drive systematic test coverage rather than ad hoc test-writing.
• Design human-in-the-loop review workflows for low-confidence or high-stakes outputs, including sampling strategy, escalation paths, and feedback loops that route confirmed failures back into evaluation datasets.
• Convert every significant production incident into permanent evaluation coverage: identify why existing evaluations didn't catch it, close the gap, and confirm the fix before redeployment.
• Establish evaluation datasets, judges, rubrics, and regression baselines as shared, versioned infrastructure usable by other project teams, rather than infrastructure rebuilt per project.
• Produce audit-ready documentation of evaluation methodology, results, and drift trends to support compliance and clinical-safety review.

Qualifications

Required

• Bachelor's degree in Computer Science, Machine Learning, or related technical field, plus 3+ years of software engineering experience with 1+ years specifically building, evaluating, or productionizing LLM or agentic AI systems or equivalent experience.
• Direct, hands-on experience integrating LLM APIs (OpenAI, Anthropic, Gemini, or similar) into production systems — this must be applied AI/ML engineering experience, not testing AI features from a pure QA viewpoint.
• Practical experience with evaluation frameworks or techniques such as RAGAS, DeepEval, LLM-as-judge scoring, or comparable evaluation tooling.
• Strong Python engineering skills, including experience with structured-output/function-calling patterns and prompt versioning.

Preferred

• Experience with agent orchestration frameworks such as Google's Agent Development Kit (ADK), LangChain/LangGraph, or similar.
• Cloud platform experience — GCP (Vertex AI, Cloud Functions), AWS (Lambda, Bedrock), or Azure (AI Foundry) — is a strong plus, particularly for serverless AI workloads.
• Experience evaluating RAG pipelines or multi-agent systems in a regulated or safety-critical domain (healthcare, finance).
• Familiarity with HIPAA/PHI handling requirements and healthcare data governance.
• Experience designing failure taxonomies, trajectory/handoff test scenarios, and severity-tiered quality gates for agentic systems.
• Exposure to observability/tracing tooling for multi-step AI pipelines (e.g., LangSmith, Datadog LLM Observability, or equivalent).

About the AI Evaluation Practice

Our AI Evaluation practice sits at the point where AI ambition meets AI accountability — every agent we ship, every model we adopt, and every prompt we change passes through the systems this role builds. If you've been in the room building and debugging LLM systems and want to make that discipline rigorous and repeatable at scale, this is that role.