AI Quality Assurance Engineer

ThirdEye Data Inc.

Bengaluru

Hybrid

INR 1,800,000 - 3,200,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

ThirdEye Data Inc. is seeking a data‑driven expert to own the evaluation framework for its AI agent system in Bengaluru. You will design and run evaluation suites, establish KPIs, and build automated pipelines that gate model and prompt changes before release.

You will measure task accuracy, tool-use correctness, latency, and cost across multiple modalities, and work with engineers to reproduce failures and implement fixes.

Qualifications

  • 4+ years in evaluation of LLM systems and agents.
  • Hands-on skill with evaluation tools and frameworks.
  • Ability to define KPIs and standardize evaluation across teams.

Responsibilities

  • Own the evaluation process for the AI agent system and its components.
  • Build and maintain evaluation suites, datasets, and regression tests.
  • Create KPIs, metrics definitions, and standardized report formats.
  • Develop evaluation pipelines and ensure changes pass before release.
  • Analyze failures, reproduce issues with engineers, and drive fixes.

Skills

LLM evaluation
Python
Test harnesses
CI/CD
Tracing & observability

Tools

RAGAs
Promptfoo
DeepEval
LangSmith
W&B Weave
OpenAI Evals

Job description

You will own the evaluation process for our AI agent system. The system includes models, prompts, tools, memory, and orchestration logic. You will test all of these parts.

You will build the tests that decide if an agent is ready to ship. You will find regressions before customers find them. You will measure agent quality with numbers, not opinions.

What You Will Do

Own the evaluation process

  • Build and maintain evaluation suites for the full agent stack.
  • Test prompt behavior, tool selection, tool calls, multi-step plans, memory use, error recovery, and task results.
  • Define quality metrics for each capability. Examples: task success rate, trajectory correctness, tool-call accuracy, latency, and cost.
  • Build and maintain golden datasets, regression suites, and adversarial test sets. Put these datasets under version control.

Build evaluation infrastructure

  • Build automated evaluation pipelines. Connect them to CI/CD.
  • Make sure each change to a model, prompt, tool, or the harness must pass evaluation before release.
  • Use more than one grading method: rule-based checks, exact match, semantic match, human review, and LLM-as-judge.
  • Calibrate LLM judges against human labels. Monitor the judges for drift.
  • Add tracing and logs to agent runs. Record each step, each tool call, and each cost. Make failures easy to diagnose.

Establish and standardize KPIs and metrics

  • Define the KPI set for agent quality. Include accuracy, faithfulness, answer relevance, context precision, context recall, tool-call correctness, hallucination rate, latency, and cost.
  • Write standard definitions for each metric. Write standard rubrics and report formats.
  • Make sure teams can compare results across models, prompts, and releases.
  • Extend evaluation to more than one modality: text, image, audio, and structured documents. Select the correct grading method for each modality.

Analyze failures and drive fixes

  • Do structured error analysis. Group failures by type.
  • Find the source of each failure: the model, the harness, or a tool.
  • Set sample sizes, pass thresholds, and confidence bounds. Agent runs are not deterministic. Plan for variance.
  • Work with engineers to reproduce failures and to verify fixes.

Protect the evaluation results

  • Find and prevent test contamination, overfitting to benchmarks, and metric gaming.
  • Keep a mix of offline evaluations, staged evaluations, and production monitoring.
  • Make sure evaluation results predict production behavior.

Set the quality bar

  • Define release criteria for agent changes. Approve or block releases with data.
  • Write clear documentation for the evaluation methods.
  • Report results to engineering and to leadership.

What We Require

AI and agent evaluation (4+ years)

  • 4 or more years of hands‑on work in evaluation of LLM systems and agents.
  • Hands‑on skill with RAGAs, Promptfoo, and DeepEval. Skill with Braintrust, LangSmith, W&B Weave, or OpenAI Evals is a plus.
  • Proof that you can create KPIs and metrics from zero: select the metrics, set the baselines, set the thresholds, and get agreement from stakeholders.
  • Proof that you can standardize evaluation across teams: shared metric definitions, reusable templates, versioned datasets, and one report format.
  • Experience with evaluation in more than one modality: text plus image, audio, video, or structured documents.
  • Knowledge of the agent evaluation problem space: trajectory evaluation, outcome evaluation, tool‑use correctness, multi‑turn behavior, non‑determinism, and prompt sensitivity.
  • Experience with LLM‑as‑judge design: rubric design, calibration against human labels, and bias control.
  • Knowledge of statistics for evaluation: sampling, significance tests, inter‑rater agreement, and variance across runs.
  • Strong Python skills. Experience with test harnesses, data pipelines, or evaluation tools that you built.
  • Skill with tracing and observability for agent systems. Example: OpenTelemetry‑style traces and structured logs of tool calls.

Core software QA

  • Knowledge of QA fundamentals: test plans, test‑case design, functional tests, regression tests, integration tests, and defect management.
  • Hands‑on skill with test automation frameworks. Examples: pytest, Playwright, Selenium.
  • Hands‑on skill with API tests. Examples: Postman, REST‑assured.
  • Experience with test suites in CI/CD pipelines. Examples: GitHub Actions, Jenkins, GitLab CI.
  • Skill with bug tracking and test management tools. Examples: Jira, TestRail, Xray.
  • Ability to write clear defect reports that engineers can reproduce.
  • Knowledge of non‑functional tests: performance, load, and reliability.

What Is a Plus

  • Experience with red‑teaming, safety evaluations, or adversarial tests of AI systems.
  • Experience with human annotation work: labeling guidelines, calibration sessions, and inter‑annotator agreement.
  • Background in CI/CD, infrastructure‑as‑code, or platform engineering.
  • Contributions to open‑source evaluation tools. Published work on evaluation methods.

What Success Looks Like

  • 90 days: You know the current agent harness. You found the gaps in the current evaluations. A baseline regression suite runs on each change.
  • 6 months: Evaluation results are trusted release gates. Failure types are tracked and decrease. Engineers come to you before they ship.
  • 12 months: Other teams use the evaluation platform on their own. Offline results predict production quality. Our quality bar is a competitive advantage.

Why This Role Matters

Agents fail in ways that normal software does not. They fail silently. They fail at random. They fail on step ten of a long task. Users trust an AI product only when the evaluation is rigorous. You own that rigor.

Job Category

Full Time

Bring Your Data or AI Vision. Let's Build It Together.

From a specific use case to a full-scale modernization, share your requirements, and our engineers will take it from there. We typically respond within 24 hours with a transparent, detailed assessment of what's possible for your business.

Who We Are

We are a full-stack AI development company that helps enterprises make better decisions, reduce costs, and operate more efficiently.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Evaluation & Reliability Engineer
Senior AI Evaluation & Reliability Engineer

Aubergine Solutions Pvt. Ltd. • Ahmedabad District

On-site
INR 3,000,000 - 6,000,000
Great Place To Work certified
Aubergine Solutions - Senior AI Evaluation & Reliability Engineer
Aubergine Solutions - Senior AI Evaluation & Reliability Engineer

Aubergine Solutions • Ahmedabad

On-site
INR 700,000 - 1,200,000
Aubergine Solutions - Senior AI Evaluation Andamp; Reliability Engineer
Aubergine Solutions - Senior AI Evaluation Andamp; Reliability Engineer

Aubergine Solutions • Ahmedabad

On-site
INR 2,800,000 - 4,500,000
AI Evaluation Engineer
AI Evaluation Engineer

Vinyl Equity, Inc. • Mangaluru

On-site
INR 1,200,000 - 1,800,000
Competitive compensation
Equity participation
Health insurance
+1
QA AI Evaluator (AI testing)
QA AI Evaluator (AI testing)

BT Group • Bengaluru

Hybrid
INR 1,500,000 - 2,400,000
Senior Research Scientist, Agent Evaluation
Senior Research Scientist, Agent Evaluation

Armis • Hyderabad, Ahmedabad

On-site
INR 2,000,000 - 4,000,000
AI QA Engineer
AI QA Engineer

Konstant Infosolutions Pvt. Ltd. • Jaipur

On-site
INR 900,000 - 1,500,000
AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Mployee.me • Mumbai

On-site
INR 1,200,000 - 1,800,000
Senior AI Quality Engineer
Senior AI Quality Engineer

Dataart Technologies • Bengaluru

On-site
INR 1,200,000 - 2,400,000
AI Engineer
AI Engineer

OnIT India • Pune District

On-site
INR 1,500,000 - 2,500,000