Senior Multimodal AI Evaluation Engineer

Luma AI; San Francisco, New York (CA, NY); On-site; USD 170,000 - 210,000; Full time

Job summary

Luma AI is seeking a Research Engineer to design and scale the infrastructure powering model evaluation efforts. You will build pipelines, metrics, and automated systems that connect model output, evaluation, and improvement across research, engineering, and product teams.

Ideal candidates have 5+ years in ML evaluation, strong Python skills, and experience with visual data and multimodal models. The role emphasizes scalable systems, CI/CD, and measurement-driven development.

Qualifications

• Master's or PhD in Computer Science, Machine Learning, or a related technical field (or equivalent industry experience).
• 5+ years of experience building ML evaluation systems, model pipelines, or large-scale infrastructure.
• Hands-on experience working with visual data (images and/or video), including evaluation, modeling, or data preparation.
• Proficiency in Python and ML frameworks (PyTorch, JAX, or TensorFlow).
• Familiarity with human-in-the-loop evaluation workflows and how to scale them with automation.
• Strong background in machine learning, with experience in generative models (diffusion, LLMs, multimodal architectures).
• Strong software engineering skills (CI/CD, testing, data pipelines, distributed systems).

Responsibilities

• Design and implement scalable pipelines for automated evaluation of generative models, with a focus on visual and multimodal outputs (image, video, text, audio).
• Develop novel metrics and evaluation models that capture qualities like fidelity, coherence, temporal consistency, and alignment with human intent.
• Integrate evaluation signals into training loops (including reinforcement learning and reward modeling) to continuously improve model performance.
• Build infrastructure for large-scale regression testing, benchmarking, and monitoring of multimodal generative models.
• Collaborate with researchers running human studies to translate human evaluation frameworks into automated or semi-automated systems.
• Partner with model researchers to identify failure cases and build targeted evaluation harnesses.
• Maintain dashboards, reporting tools, and alerting systems to surface evaluation results to stakeholders.
• Stay current with emerging evaluation techniques in generative AI, multimodal LLMs, and perceptual quality assessment.

Skills

Python

CI/CD

Distributed systems

ML frameworks (PyTorch/JAX/TF)

Education

Master's or PhD in Computer Science, ML, or related field

Tools

PyTorch

JAX

TensorFlow

Job description

Luma AI is seeking a Research Engineer to design and scale the infrastructure powering model evaluation efforts. You will build pipelines, metrics, and automated systems that connect model output, evaluation, and improvement across research, engineering, and product teams.

Ideal candidates have 5+ years in ML evaluation, strong Python skills, and experience with visual data and multimodal models. The role emphasizes scalable systems, CI/CD, and measurement-driven development.