Senior AI Evaluation & RLHF Specialist

Innodata India; Dadri; On-site; INR 1,800,000 - 2,800,000; Full time

Job summary

Innodata India seeks an experienced AI Evaluation and RLHF Specialist to join our high-complexity alignment team. You will generate high-quality evaluation datasets, perform side-by-side model comparisons, and craft actionable feedback to improve reward model performance.

The role requires strong English writing, hands-on data annotation experience, and the ability to adapt to evolving guidelines in fast-paced sprints. Opportunity to mentor junior raters and lead calibration sessions.

Qualifications

• Hands-on data annotation or evaluation on AI platforms.
• Mastery in multi-dimensional rubric scoring and chain-of-thought verification.
• Excellent written English with clear, objective rationales under tight SLAs.
• Adaptability to rapidly evolving project guidelines and fast-paced sprints.

Responsibilities

• Author high-fidelity preference datasets and rank model outputs.
• Conduct Side-by-Side evaluations across factuality, helpfulness, harmlessness, and coherence.
• Rewrite sub-optimal outputs and provide corrective learning signals.
• Evaluate step-by-step reasoning and tool-use trajectories of autonomous agents.
• Create golden sets and worked examples for training calibration cohorts.
• Lead daily IRR calibration and drive guideline iterations.

Skills

AI RLHF evaluation

Data annotation

English communication

Task calibration

Education

STEM degree

Tools

Python scripting

Prompt engineering

Job description

ROLE OVERVIEW & OBJECTIVE:

We are expanding our core evaluation and are looking to hire experienced AI Evaluation and Reinforcement Learning from Human Feedback (RLHF) Specialists into our high-complexity alignment pipeline. This cohort directly powers reward model training, preference modeling, complex multi-turn reasoning, and critique-and-rework loops for top-tier frontier models. We are sourcing experienced practitioners from leading AI data platforms who require zero ramp-up time, demonstrate rigorous adherence to dynamic rubrics, and deliver ironclad, defensible evaluation rationales.

KEY RESPONSIBILITIES & CORE WORKFLOWS:

• High-Precision Preference Data Generation: Author high-fidelity preference datasets and rank multi-turn model completions to optimize reward model loss functions.
• Side-by-Side Evaluations: Execute deep SxS comparative evaluations across factuality, helpfulness, harmlessness, logical coherence, and instruction-following.
• Improve Model output: Rewrite sub-optimal model outputs and break down complex chain-of-thought failures into structured corrective learning signals.
• Multi-Step Reasoning & Agentic Auditing: Evaluate step-by-step mathematical, logical, and tool-use trajectories generated by autonomous AI agents.
• Golden Sets & Worked Examples: Construct reference 'golden responses' and edge-case worked examples to train and calibrate downstream evaluation cohorts.
• Inter-Rater Reliability (IRR) Leadership: Participate in daily calibration huddles, driving consensus on ambiguous prompts and rapid guideline iterations.

Mandatory Requirements:

• Direct AI Platform Experience: 2+ years of hands-on data annotation, model evaluation, or RLHF preference scoring on recognized platforms
• Complex Task Track Record: Demonstrable mastery in multi dimensional rubric scoring, and chain-of-thought verification.
• Written Communication: English proficiency with proven discipline in articulating clear, fact-backed, objective rationales under tight SLAs.
• Operational Agility: High tolerance for rapidly evolving project guidelines, shifting evaluation criteria, and fast-paced delivery sprints.

Preferred Qualifications:

• Domain Specialization: Advanced educational background in STEM, Law, Finance, Medicine, or Humanities.
• Technical Literacy: Basic Python scripting, prompt engineering knowledge, or familiarity with loss functions and reward modeling concepts.
• Leadership Experience: Prior experience mentoring junior raters, conducting secondary QA audits, or serving as a task calibration lead.