Remote LLM Evaluator: Model Response Analyst

Odixcity Consulting; La Réunion; Remote; EUR 49,000 - 80,000; Full time

Job summary

Odixcity Consulting is seeking an LLM Evaluator to assess and improve large language model outputs. You will analyze factuality, safety, coherence, and instruction adherence, and provide actionable feedback to model training teams.

The role focuses on evaluating model responses, breaking prompts for safety testing, and ensuring consistent scoring across a global team. Remote work and collaboration with QA are expected.

Qualifications

• Minimum of 2 years of professional experience in NLP/AI related field.
• Bachelor’s degree in Computer Science or related field.
• Deep understanding of prompt engineering and model behavior testing.

Responsibilities

• Evaluate and rank model-generated text using rubrics for factuality, coherence, safety, and adherence to guidelines.
• Review multiple model outputs and justify preferred responses to human evaluators.
• Provide concise feedback to model training teams on recurring failures and edge cases.
• Identify and flag novel model behaviors for the research team to document.
• Calibrate scoring with cross-check sessions to ensure inter-rater reliability across the global team.

Job description

Odixcity Consulting is seeking an LLM Evaluator to assess and improve large language model outputs. You will analyze factuality, safety, coherence, and instruction adherence, and provide actionable feedback to model training teams.

The role focuses on evaluating model responses, breaking prompts for safety testing, and ensuring consistent scoring across a global team. Remote work and collaboration with QA are expected.