Multimodal AI Evaluation Engineer

lumalabs-ai; New York (NY); On-site; USD 190,000 - 375,000; Full time

Job summary

Luma AI is seeking a Research Engineer in New York to design and scale infrastructure powering model evaluation for multimodal outputs. You will build pipelines, metrics, and automated systems that close the loop between model output, evaluation, and improvement, collaborating across research, engineering, and product teams.

Responsibilities include designing scalable evaluation pipelines, developing metrics for fidelity and temporal coherence, integrating signals into training loops (RL and

Qualifications

• Master's or PhD in CS/ML or related field; equivalent industry experience.
• 5+ years building ML evaluation systems or large-scale infrastructure.
• Hands-on experience with visual data (images/videos) and evaluation.
• Proficiency in Python and ML frameworks (PyTorch, JAX, or TensorFlow).
• Familiarity with human-in-the-loop evaluation workflows and automation.
• Strong ML background with generative models (diffusion, LLMs, multimodal).
• Strong software engineering skills (CI/CD, data pipelines, distributed systems).

Responsibilities

• Design scalable pipelines for automated evaluation of multimodal outputs (image, video, text, audio).
• Develop metrics and evaluation models that capture fidelity, coherence, temporal consistency, and alignment with human intent.
• Integrate evaluation signals into training loops (including reinforcement learning and reward modeling) to improve model performance.
• Build infrastructure for large-scale regression testing, benchmarking, and monitoring of multimodal generative models.
• Collaborate with researchers to translate human evaluation frameworks into automated systems.
• Partner with model researchers to identify failure cases and build evaluation harnesses.
• Maintain dashboards, reporting tools, and alerting systems to surface evaluation results to stakeholders.
• Stay current with emerging evaluation techniques in generative AI, multimodal LLMs, and perceptual quality assessment.

Skills

ML evaluation systems

Model pipelines

Large-scale infrastructure

Python programming

Education

Master's or PhD in CS/ML

Tools

PyTorch

JAX

TensorFlow

Job description

Luma AI is seeking a Research Engineer in New York to design and scale infrastructure powering model evaluation for multimodal outputs. You will build pipelines, metrics, and automated systems that close the loop between model output, evaluation, and improvement, collaborating across research, engineering, and product teams.

Responsibilities include designing scalable evaluation pipelines, developing metrics for fidelity and temporal coherence, integrating signals into training loops (RL and