AI Evaluation Architect

Nouvells; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Nouvells seeks an AI Evaluation Specialist to lead quality, safety, and qualitative benchmarking of its models and agentic workflows. You will design and execute evaluation frameworks that measure nuanced LLM performance, detect hallucinations, assess multi-step reasoning, and test conversational quality.

This role sits at the intersection of technical operations, linguistics, and ML engineering, translating business expectations into repeatable evaluation suites and bridging human evaluation

Qualifications

• Bachelor's Degree or equivalent practical experience in a quantitative/technical field.
• Experience translating business expectations into evaluation benchmarks is a plus.

Responsibilities

• Design and maintain evaluation benchmark suites across core metrics like factual accuracy, hallucinations, style adherence, latency, and context retention.
• Create golden datasets and adversarial test sets for edge cases and multi-turn context drift.
• Manage qualitative review cycles for model versions and prompt iterations before deployment to production.
• Build automated evaluation workflows using LLMs as judges and validate correlation with human consensus.
• Audit annotation outputs for inter-annotator agreement and calibrate scoring discrepancies.

Education

Bachelor's Degree or equivalent practical experience in a quantitative/technical field

Job description

Nouvells seeks an AI Evaluation Specialist to lead quality, safety, and qualitative benchmarking of its models and agentic workflows. You will design and execute evaluation frameworks that measure nuanced LLM performance, detect hallucinations, assess multi-step reasoning, and test conversational quality.

This role sits at the intersection of technical operations, linguistics, and ML engineering, translating business expectations into repeatable evaluation suites and bridging human evaluation