Senior Research Scientist, Agent Evaluation

Armis

Hyderabad, Ahmedabad

On-site

INR 2,000,000 - 4,000,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Armis is building the evaluation layer that validates AI agents before they reach customers. You will own core parts of the platform, the pipeline that runs traces through model-based judges at scale, and the scoring logic that turns raw output into actionable results for teams.

This role emphasizes end-to-end ownership, hands-on production work with LLM APIs, and shaping methodologies for agent reasoning, planning, and tool use. Strong Python skills and clear communication are essential.

Qualifications

  • 5+ years in ML, applied AI, prompt engineering, agentic AI, including shipping something real to users.
  • Strong Python skills.
  • Practical depth in agentic AI and context engineering: planning, reasoning, memory, tool use, retrieval, long-context.
  • Experience designing evaluation methodologies, not just running evaluations.
  • Hands-on production work with LLM APIs - prompt engineering, structured output, cost and latency tradeoffs.
  • Clear communication with technical and non-technical audiences.
  • Nice to have Experience with AI-assisted development tools (Claude code, Windsurf, or similar).

Responsibilities

  • Design evaluation methodologies and benchmarks for agent reasoning, planning, tool use, reliability, and safety - across LLM-as-a-Judge, trajectory-based, and human evaluation
  • Take problems from research question to prototype to shipped feature, owning them end to end
  • Build and harden the pipelines and scoring logic behind customer-facing evaluation
  • Curate synthetic and real-world datasets; measure the evaluator itself for consistency and agreement with human labels Qualifications

Skills

Machine Learning
Python
Agentic AI
Context Engineering
Prompt Engineering
Communication
LLM APIs

Tools

Claude Code
Windsurf

Job description

Job Description

We build the evaluation layer that validates AI agents before they reach customers - an automated system that scores across large volumes of agent traces. Youll own core parts of that platform: the pipeline that runs traces through model-based judges at scale, and the scoring logic that turns raw output into results teams can act on.

What youll do
  • Design evaluation methodologies and benchmarks for agent reasoning, planning, tool use, reliability, and safety - across LLM-as-a-Judge, trajectory-based, and human evaluation
  • Take problems from research question to prototype to shipped feature, owning them end to end
  • Build and harden the pipelines and scoring logic behind customer-facing evaluation
  • Curate synthetic and real-world datasets; measure the evaluator itself for consistency and agreement with human labels Qualifications
What were looking for
  • 5+ years in ML, applied AI, prompt engineering, agentic AI, including shipping something real to users
  • Strong Python skills
  • Practical depth in agentic AI and context engineering: planning, reasoning, memory, tool use, retrieval, long-context
  • Experience designing evaluation methodologies, not just running evaluations
  • Hands-on production work with LLM APIs - prompt engineering, structured output, cost and latency tradeoffs
  • Clear communication with technical and non-technical audiences
  • Good to have Experience with AI-assisted development tools (Claude code, Windsurf, or similar)
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Research Scientist, Agent Evaluation
Senior Research Scientist, Agent Evaluation

ServiceNow • Telangana

Hybrid
INR 2,500,000 - 4,500,000
AI Quality Assurance Engineer
AI Quality Assurance Engineer

ThirdEye Data Inc. • Bengaluru

Hybrid
INR 1,800,000 - 3,200,000
Aubergine Solutions - Senior AI Evaluation & Reliability Engineer
Aubergine Solutions - Senior AI Evaluation & Reliability Engineer

Aubergine Solutions • Ahmedabad

On-site
INR 700,000 - 1,200,000
AI Engineer: Agents & Evaluation
AI Engineer: Agents & Evaluation

WOW Softech • Gurugram District

On-site
INR 1,400,000 - 2,100,000
Senior AI Evaluation & Reliability Engineer
Senior AI Evaluation & Reliability Engineer

Aubergine Solutions Pvt. Ltd. • Ahmedabad District

On-site
INR 3,000,000 - 6,000,000
Great Place To Work certified
AI Evaluation Engineer
AI Evaluation Engineer

Vinyl Equity, Inc. • Mangaluru

On-site
INR 1,200,000 - 1,800,000
Competitive compensation
Equity participation
Health insurance
+1
Senior Research Scientist, Agent Evaluation
Senior Research Scientist, Agent Evaluation

ServiceNow • Hyderabad

Hybrid
INR 4,000,000 - 8,000,000
Aubergine Solutions - Senior AI Evaluation Andamp; Reliability Engineer
Aubergine Solutions - Senior AI Evaluation Andamp; Reliability Engineer

Aubergine Solutions • Ahmedabad

On-site
INR 2,800,000 - 4,500,000
Senior AI Quality Engineer
Senior AI Quality Engineer

Dataart Technologies • Bengaluru

On-site
INR 1,200,000 - 2,400,000
Senior AI Engineer — Agentic Systems
Senior AI Engineer — Agentic Systems

Glorious Insight Pvt. Ltd. • India

On-site
INR 4,000,000 - 7,000,000