Senior Linguist (Text LLM – Model Evaluation)

BharatGen; Mumbai; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

A leading tech firm in Mumbai is seeking a Text LLM Model Evaluation Lead to oversee the evaluation processes of their language models. You will design evaluation frameworks and collaborate with engineers and linguists to ensure quality assessments across tasks and languages. Ideal candidates have a Master's or PhD with strong experience in NLP, proficiency in Python, and a passion for refining evaluation methodologies. This role contributes significantly to the advancement of multilingual NLP capabilities.

Qualifications

• 3+ years of experience in NLP or GenAI projects with evaluation exposure.
• Experience managing multi-language annotation or evaluation projects is preferred.
• Familiarity with bias detection, toxicity analysis, or fairness evaluation.

Responsibilities

• Design and manage evaluation frameworks for text-based large language models.
• Conduct error and trend analysis on model outputs.
• Train and mentor junior linguists in evaluation techniques.

Skills

Human evaluation framework design

Analytical reporting

Collaboration with ML teams

Multilingual evaluation projects

Bias detection awareness

Education

Master's or PhD in Linguistics/Computational Linguistics

Tools

Python

LangGraph

DSPy

AutoGen

CrewAI

Job description

Job Summary

The Text LLM Model Evaluation Lead will own the end-to-end process of evaluating BharatGen’s text-based large language models. You will design human evaluation frameworks, test sets, rubrics, and metrics that assess model outputs across multiple tasks and languages. Working closely with ML engineers, linguists, and data operations, you’ll ensure that every model iteration is measured with rigor, fairness, and linguistic precision.

Key Responsibilities

• Design and manage evaluation frameworks for BharatGen’s Text LLM, covering diverse tasks such as summarization, dialogue, question answering, reasoning.
• Define evaluation dimensions (coherence, factuality etc.).
• Develop human evaluation rubrics, task-specific test sets for multiple languages.
• Establish evaluation workflows using human judgment that complement automated metrics.
• Create documentation, checklists, and SOPs to ensure replicability of evaluations across model versions.
• Collaborate with the Data Ops Manager to execute large-scale human evaluations across languages, align throughput, timelines, and cost controls.
• Review and refine annotation guidelines to ensure inter-annotator consistency.
• Design sampling and spot-checking methods for maintaining high data integrity.
• Implement inter-annotator agreement tracking, quality audits.
• Analytical Evaluation & Reporting:
• Conduct error and trend analysis on model outputs across evaluation rounds.
• Interpret results to highlight strengths, regressions, or recurring weaknesses.
• Present findings and recommendations to ML engineers and leadership in structured, data-backed reports.
• Collaborate with the ML team to refine models based on evaluation results and feedback loops.
• Metrics & Tooling:
• Identify or adapt suitable automatic evaluation metrics (e.g., BLEU, ROUGE, BERTScore, toxicity classifiers, etc.) to complement human evaluation.
• Use simple scripts/dashboards to track scores, trends, & evaluation throughput.
• Partner with Data Ops and product engineers to improve internal tools for managing evaluation tasks and results visualization.
• Cross-functional Collaboration:
• Train and mentor junior linguists in designing high-quality evaluation schemes.
• Participate in design reviews to ensure evaluation insights are integrated into model training and product goals.
• Master’s or PhD in Linguistics/Computational Linguistics with 3+ years of experience working on NLP or GenAI projects, with exposure to model evaluation, test-set design, or linguistic quality assessment.
• Proficiency in Python and agentic frameworks such as LangGraph, DSPy, AutoGen, CrewAI, etc.
• Experience collaborating with ML or data science teams on evaluation or model analysis workflows.
• Experience managing multi-language annotation or evaluation projects is preferred.
• Experience with multilingual LLMs or Indian language NLP.
• Exposure to instruction-tuning, safety evaluation, or RLHF workflows.
• Familiarity with bias detection, toxicity analysis, or fairness evaluation.
• Prior experience training or mentoring annotation/evaluation teams.