AI Evaluation Specialist

Nouvells; United States; A distancia; USD 90.000 - 130.000; Jornada completa

Descripción de la vacante

Nouvells seeks an AI Evaluation Specialist to lead quality, safety, and qualitative benchmarking of its models and agentic workflows. You will design and execute evaluation frameworks that measure nuanced LLM performance, detect hallucinations, assess multi-step reasoning, and test conversational quality.

This role sits at the intersection of technical operations, linguistics, and ML engineering, translating business expectations into repeatable evaluation suites and bridging human evaluation

Formación

• Bachelor's Degree or equivalent practical experience in a quantitative/technical field.
• Experience translating business expectations into evaluation benchmarks is a plus.

Responsabilidades

• Design and maintain evaluation benchmark suites across core metrics like factual accuracy, hallucinations, style adherence, latency, and context retention.
• Create golden datasets and adversarial test sets for edge cases and multi-turn context drift.
• Manage qualitative review cycles for model versions and prompt iterations before deployment to production.
• Build automated evaluation workflows using LLMs as judges and validate correlation with human consensus.
• Audit annotation outputs for inter-annotator agreement and calibrate scoring discrepancies.

Educación

Bachelor's Degree or equivalent practical experience in a quantitative/technical field

Descripción del empleo

Nouvells is seeking AI Evaluation Specialists who can assess the quality, accuracy, relevance, safety, and usefulness of AI generated outputs.You'll help organizations understand how effectively AI systems perform in real-world scenarios and identify opportunities for improvement.

As our AI Evaluation Specialist, you will own the quality, safety, and qualitative benchmarking of Nouvell's models and agentic workflows. You will design and execute evaluation frameworks that measure nuanced LLM performance - detecting subtle hallucinations, evaluating multi-step reasoning, testing conversational quality, and running safety stress-tests.This role sits at the intersection of technical operations, linguistics, and machine learning engineering. You will translate qualitative business expectations into programmatic benchmarks and repeatable evaluation suites, bridging human evaluation with automated LLM-as-a-judge pipelines.

Key responsibilities

• Design and maintain comprehensive, domain-specific evaluation benchmark suites across core metrics: factual accuracy, hallucination rates, style adherence, latency, and context retention.
• Create curated "golden datasets" and adversarial test sets that challenge models on edge cases, multi-turn context drift, and ambiguity.
• Manage qualitative review cycles for new model versions, prompt iterations, and fine-tuned checkpoints before deployment to production.
• Build automated evaluation workflows (LLM-as-a-judge) and validate that automated scoring closely correlates with human consensus.
• Audit annotation outputs for inter-annotator agreement (IAA) using statistical measures (e.g., Cohen's kappa) and calibrate scoring discrepancies.

Qualifications

• Bachelor's Degree
• Equivalent practical experience: A degree from an accredited institution in a relevant quantitative, technical, or analytical field