AI Evaluations Engineer — Benchmarking Frontiers

Meta; Menlo Park (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Meta is seeking Research Engineers to join the Evaluations team within Meta Superintelligence Labs. You will curate and build benchmarks for our advanced AI models across text, vision, audio, and beyond, collaborating with world-class researchers to deploy novel evaluation environments.

This is a highly technical role requiring practical research engineering skills and independence. You will design and implement scalable evaluation pipelines, source data, and contribute to tooling that measures

Qualifications

• Bachelor's degree in Computer Science, Computer Engineering, or related field, or equivalent practical experience.
• 4+ years of experience in machine learning engineering, machine learning research, or related technical role.
• Proficiency in Python and experience with ML frameworks such as PyTorch.
• Experience identifying, designing and completing medium to large technical features independently, without guidance.
• Demonstrated experience in software engineering practices including version control, testing, and code review practices.
• Publications at peer-reviewed venues (NeurIPS, ICML, ICLR, ACL, EMNLP, or similar) related to language model evaluation, benchmarking, or deep learning.
• Hands-on experience with language model post-training and deep learning systems, or building reinforcement learning environments.
• Experience implementing or developing evaluation benchmarks for large language models and multimodal models (e.g., vision-language, audio, video).
• Experience working with large-scale distributed systems and data pipelines.
• Familiarity with language model evaluation frameworks and metrics.
• Track record of open-source contributions to ML evaluation tools or benchmarks.

Responsibilities

• Curate and integrate publicly available and internal benchmarks to direct the capabilities of frontier model development.
• Develop and implement evaluation environments, including environments for novel model capabilities and modalities.
• Collaborate with external data vendors to source and prepare high-quality evaluation datasets.
• Execute on the technical vision of research scientists designing new benchmarks and evaluations.
• Build robust, reusable evaluation pipelines that scale across multiple model lines and product areas.
• Contribute to evaluation tooling that measures the quality and reliability of evaluation suites.

Skills

Python

ML frameworks

Independent work

Software engineering practices

Publications in peer venues

Reinforcement learning environments

Benchmark development

Multimodal models

Distributed data pipelines

Education

Bachelor's degree in Computer Science/Engineering or equivalent

Tools

PyTorch

Job description

Meta is seeking Research Engineers to join the Evaluations team within Meta Superintelligence Labs. You will curate and build benchmarks for our advanced AI models across text, vision, audio, and beyond, collaborating with world-class researchers to deploy novel evaluation environments.

This is a highly technical role requiring practical research engineering skills and independence. You will design and implement scalable evaluation pipelines, source data, and contribute to tooling that measures