Remote LLM Evaluator: Model Response Analyst

Odixcity Consulting; South Africa; Remote; ZAR 360,000 - 540,000; Full time

Job summary

Odixcity Consulting seeks a detail-oriented LLM Evaluator to assess and improve large language models. You will analyze model outputs for factuality, coherence, safety, and alignment with guidelines, providing actionable feedback to ML teams.

You will review multiple responses, justify preferences, and help calibrate scoring with global evaluators, contributing to a living library of model behaviors and failure modes.

Qualifications

• Minimum 2 years of professional experience in NLP/AI-related fields.
• Bachelor’s degree in CS or related field.
• Ability to explain why text is “good” or “bad” based on logic, tone, factuality and adherence to guidelines.
• Experience with RLHF data collection.
• Experience coordinating evaluation teams and IAA calibration.

Responsibilities

• Evaluate and rank model outputs using rubrics covering factuality, coherence, safety, instruction-following, and creativity.
• Review multiple responses to a prompt and justify preferred output with clear reasoning.
• Provide feedback to ML teams on recurring failures observed during evaluation.
• Craft prompts to challenge models and reveal safety vulnerabilities.
• Collaborate to update evaluation guidelines for edge cases and ambiguities.
• Participate in cross-checking sessions to calibrate scoring across evaluators.
• Analyze root causes of errors beyond surface scores.
• Flag novel model behaviors to researchers for library of outputs and failure modes.

Skills

Computational Linguistics

Data Analysis

Technical Writing

Quality Assurance (NLP/AI)

Cognitive Science

Prompting & Prompt Design

RLHF Data Collection

Education

Bachelor’s degree in Computer Science or related field

Job description

Odixcity Consulting seeks a detail-oriented LLM Evaluator to assess and improve large language models. You will analyze model outputs for factuality, coherence, safety, and alignment with guidelines, providing actionable feedback to ML teams.

You will review multiple responses, justify preferences, and help calibrate scoring with global evaluators, contributing to a living library of model behaviors and failure modes.