Lead Data Annotation & AI Evaluation Specialist

Objectways; Chennai District; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

Objectways is seeking a highly technical Lead Data Annotation & AI Evaluation Specialist with at least 5 years of hands-on experience in complex data annotation, AI/ML data operations, or model evaluation. The role involves leading technically complex AI data projects with LLMs, multimodal data, NLP, and model evaluation, while ensuring clear workflows and QA criteria.

The ideal candidate will read detailed customer specifications end‑to‑end, design annotation workflows, and guide an annotation

Qualifications

• 5+ years hands-on experience in data annotation, AI/ML data operations, or model evaluation.
• Ability to design annotation/workflow specs from customer requirements.
• Lead technically complex AI data projects involving LLMs, multimodal data, NLP, and model evaluation.
• Bridge between customer specs, engineering/ML teams, and annotation operations.

Responsibilities

• Read and interpret complex customer annotation and AI evaluation specifications.
• Create SOPs, guidelines, decision trees, examples, edge cases, and QA checklists.
• Design annotation schemas, taxonomies, labels, metadata, acceptance criteria, and review workflows.
• Understand multi-turn conversations, model behavior, tool calls, prompts, and contextual dependencies.
• Work with JSON/JSONL, structured data, APIs, tool calls, logs, and annotation platforms.
• Create gold‑standard examples and benchmark datasets before production.
• Train annotators and reviewers on complex projects and certify readiness.

Skills

LLMs & Generative AI

Multi-turn conversational systems

NLP annotation

Annotation schema design

QA and QC

Data ops for AI/ML

Tools

CVAT

Label Studio

Encord

SageMaker Ground Truth

JSON/JSONL

APIs

Job description

Experience: 5–8+ years

Notice Period : Immediate

Role Summary

Objectways is seeking a highly technical Lead Data Annotation & AI Evaluation Specialist with at least 5 years of hands‑on experience in complex data annotation, AI/ML data operations, or model evaluation.

This role will lead technically complex AI data projects involving LLMs, agentic AI, multimodal data, computer vision, NLP, model evaluation, safety evaluation, and human-in-the-loop workflows.

The ideal candidate must be able to take a detailed customer specification, read it end‑to‑end, understand the technical and operational intent, identify ambiguities and risks, design the annotation/evaluation workflow, create examples and edge cases, define QA criteria, and guide an annotation team through successful execution.

This is not simply a people‑management or annotation‑production role. We need someone who can think technically and act as the bridge between the customer specification, engineering/ML teams, and annotation operations.

Key Responsibilities

• Read and interpret complex customer annotation, data‑generation, and AI‑evaluation specifications.
• Convert requirements into clear SOPs, annotation guidelines, decision trees, examples, edge cases, and QA checklists.
• Independently determine how a project should be executed rather than waiting for step‑by‑step instructions.
• Design annotation schemas, taxonomies, labels, metadata, acceptance criteria, and review workflows.
• Understand complex multi‑turn conversations, model behavior, tool calls, system prompts, agent trajectories, and contextual dependencies.
• Work with JSON/JSONL, structured data, APIs, tool‑call traces, logs, model outputs, and annotation platforms.
• Create gold‑standard examples and benchmark datasets before production begins.
• Conduct pilot annotations personally and identify gaps in customer guidelines before scaling to the annotation team.
• Define inter‑annotator agreement, arbitration, QC sampling, error taxonomy, and acceptance thresholds.
• Analyze disagreements and distinguish annotator error, guideline ambiguity, tooling issues, and genuinely ambiguous data.
• Train annotators and reviewers on technically complex projects and certify readiness before production.
• Work closely with engineering and ML teams to understand model inputs/outputs and improve annotation tooling and workflows.
• Communicate directly with customers or internal project teams to raise technically meaningful clarification questions.
• Monitor production quality and identify systematic errors rather than simply reporting aggregate accuracy.
• Perform root‑cause analysis and recommend changes to guidelines, workflows, tooling, or training.

Technical Skills

The candidate should be comfortable with:

• LLMs and Generative AI
• AI agents and multi‑turn conversational systems
• AI safety and model behavior evaluation
• NLP annotation and classification
• Annotation schema and taxonomy design
• JSON/JSONL and structured datasets
• Tool calls, API concepts, and structured model outputs
• Python or SQL at a working level
• Regular expressions and basic scripting/data analysis
• Annotation platforms such as CVAT, Label Studio, Encord, SageMaker Ground Truth, or equivalent
• Quality metrics including accuracy, precision/recall, confusion matrices, IAA/Cohen's Kappa/Fleiss' Kappa