Senior AI Evaluation Analyst

Naviq; Ernakulam; On-site; INR 600,000 - 900,000; Full time

Job summary

Naviq is seeking an experienced, detail-oriented professional to join its Machine Learning Science team in evaluating and improving AI-powered travel products. You will review AI outputs, annotate data, and apply structured guidelines to ensure quality, suggesting actionable improvements.

The role focuses on hands-on evaluation, data analysis, and documentation, with collaboration across ML Scientists and product teams to enhance agent performance.

Qualifications

• 5–8 years of relevant experience in data annotation, data labeling, AI evaluation, content quality, QA, or similar roles.
• Strong ability to analyze data, apply guidelines, and derive actionable insights.
• Proficiency with Excel or Google Sheets for data filtering, analysis and reporting.

Responsibilities

• Evaluate AI-generated responses from conversational agents against predefined guidelines and quality standards.
• Perform and review data annotation, labeling and classification to assess accuracy, relevance and completeness.
• Identify and document incorrect, misleading, incomplete or irrelevant responses, including potential AI hallucinations.
• Conduct quality assurance reviews for consistency, accuracy, and adherence to guidelines.
• Analyze large datasets to identify recurring errors, trends, and improvement opportunities.
• Prepare clear documentation of findings, root causes, observations, and recommended improvements.
• Collaborate with ML Scientists and product teams to communicate quality insights and support improvements.
• Maintain accurate documentation and track issues through ticketing/workflow systems.

Skills

Data annotation

Quality assurance

Data analysis

Excel/Sheets

SQL (basic)

Documentation

Communication

Independent work

Tools

Excel

Google Sheets

SQL

Job description

Seeking an experienced and detail-oriented professional to join its Machine Learning Science team, supporting the development and improvement of AI-powered travel products. The team works on conversational AI agents that help travelers discover options, compare products, plan trips, and receive assistance throughout their travel journey.

You will play a key role in evaluating the quality, accuracy, relevance, and reliability of AI-generated responses against established guidelines and quality metrics. You will analyze evaluation results, identify recurring errors and quality gaps, and translate findings into actionable insights that help improve AI agent performance.

This role requires a strong foundation in data annotation, quality evaluation, analytical problem-solving, and guideline-based decision-making. The position is suited to professionals who enjoy hands-on evaluation and analysis of AI outputs rather than traditional business reporting or dashboard development.

Ideal Candidate Profile

The ideal candidate is an experienced, hands-on professional who combines strong evaluation and quality assurance skills with analytical thinking. They should be comfortable reviewing large volumes of information, applying structured guidelines, identifying recurring quality issues, and converting observations into meaningful insights.

Candidates with experience in AI evaluation, data annotation, content quality, search relevance, or similar quality-focused roles are encouraged to apply. Direct experience with LLMs or conversational AI is an advantage but is not mandatory for candidates with strong transferable experience.

Key Responsibilities

• Evaluate AI-generated responses from conversational agents against predefined guidelines, evaluation criteria, and quality standards.
• Perform and review data annotation, labelling, and classification to assess the accuracy, relevance, completeness, and appropriateness of AI-generated content.
• Identify and document incorrect, misleading, incomplete, inconsistent, or irrelevant responses, including potential AI hallucinations.
• Conduct quality assurance reviews to ensure consistency, accuracy, and adherence to evaluation guidelines.
• Analyze large and complex datasets to identify recurring errors, quality trends, performance gaps, and opportunities for improvement.
• Interpret quality metrics such as error rates, evaluation consistency, reviewer disagreement, and other relevant indicators.
• Investigate ambiguous or borderline cases and apply sound judgment while adhering to established evaluation criteria.
• Prepare clear documentation of evaluation findings, root causes, observations, and recommended improvements.
• Use Excel or Google Sheets and, where applicable, basic SQL to filter, analyze, and summarize evaluation data.
• Collaborate with Machine Learning Scientists, AI product teams, and other stakeholders to communicate quality insights and support improvements in AI agent performance.
• Contribute to the refinement of evaluation guidelines, quality frameworks, and review processes based on observed issues and emerging patterns.
• Maintain accurate documentation and track identified issues through appropriate ticketing and workflow management systems.

Required Skills and Qualifications

• 5–8 years of relevant experience in data annotation, data labeling, AI evaluation, content quality, quality assurance, search relevance evaluation, or similar guideline-based assessment roles.
• Strong experience evaluating data, content, or system outputs against predefined guidelines, quality standards, and measurable criteria.
• Demonstrated ability to analyze large datasets, identify patterns, investigate quality issues, and derive actionable insights.
• Strong analytical thinking, problem-solving, and decision-making skills, particularly when dealing with ambiguous or inconsistent information.
• Excellent attention to detail and the ability to apply evaluation criteria consistently.
• Proficiency in Microsoft Excel or Google Sheets for data filtering, analysis, quality tracking, and reporting.
• Strong written communication skills and the ability to document findings, explain issues, and communicate insights clearly.
• Ability to work independently on evaluation tasks while collaborating effectively with cross-functional teams.
• Genuine interest in artificial intelligence, conversational AI, and data-driven product improvement.
• Experience with Human-in-the-Loop (HITL) workflows, AI training data validation, model output assessment, or LLM-as-a-judge evaluation.

Good to Have

• Exposure to Large Language Models (LLMs), Generative AI, conversational agents, or AI-generated content evaluation.
• Familiarity with evaluating AI responses for factual accuracy, relevance, completeness, helpfulness, and hallucinations.
• Basic SQL knowledge for extracting, filtering, and analyzing datasets.
• Experience with quality metrics such as error rates, inter-annotator agreement, reviewer disagreement, and evaluation consistency.
• Experience maintaining evaluation guidelines, quality documentation, review workflows, or issue-tracking systems.
• Exposure to travel, hospitality, customer experience, search relevance, policy evaluation, or content quality operations.