Qualitative Evaluation Engineer: Define Believability

Luma; United States; On-site; USD 120,000 - 190,000; Full time

Job summary

Luma seeks a Qualitative Evaluation Engineer to define believability, identity retention, and scene coherence, building evaluative systems that match human perception. You will work with researchers and technical artists to steer model development beyond checkbox metrics.

In the first 90 days you will immerse, stand up a qualitative framework, and scale it into the eval loop, turning nuanced judgments into actionable feedback for fine-tuning and dataset curation.

Qualifications

• 5+ years in product evaluation, UX research, model testing, or similar qualitative assessment.
• Master's or higher in Cognitive Science, HCI, Design Research, Psychology, Media Studies, or related field.
• Deep familiarity with creative workflows for generative models (animation, filmmaking, digital art, VFX).
• Systems thinking: you can define abstract qualities like believability or scene coherence in clear evaluative terms.
• Excellent written communication and the ability to synthesize nuanced judgment into actionable insight.
• Comfort working across engineers, researchers, and creatives.

Responsibilities

• Evaluate generative model performance across diverse tasks, prompts, and modalities, and surface the failure modes, regressions, and edge cases that hurt product quality.
• Build and maintain qualitative evaluation frameworks that are scalable and reusable.
• Translate high-level product goals into concrete evaluative criteria.
• Lead qualitative studies, side-by-side comparisons, and human-in-the-loop evaluations.
• Turn nuanced judgments into clear feedback that informs fine-tuning, dataset curation, and product UX.
• Work closely with technical artists and engineers to keep evaluations aligned with model capabilities and real use cases.

Skills

Qualitative evaluation

UX research

Model testing

Written communication

Cross-functional work

Education

Master's degree or higher in Cognitive Science / HCI / Design Research / Psychology / Media Studies

Job description

Luma seeks a Qualitative Evaluation Engineer to define believability, identity retention, and scene coherence, building evaluative systems that match human perception. You will work with researchers and technical artists to steer model development beyond checkbox metrics.

In the first 90 days you will immerse, stand up a qualitative framework, and scale it into the eval loop, turning nuanced judgments into actionable feedback for fine-tuning and dataset curation.