Remote AI Model Evaluation Engineer for Top Contributors

AI Work Expert; San Francisco (CA); Hybrid; USD 83,000 - 165,000; Full time

Job summary

AI Work Expert is seeking a contractor to design and develop real-world problems for an Agentic development environment. You will implement tasks in Python, create environments, and provide a golden solution and test notebooks to evaluate model performance.

The role emphasizes strong expertise in data science, ML, coding, or frontier STEM, with a demand for reliable weekly engagement (30+ hours). Remote work is supported, and the project is focused on advancing AI evaluation tasks.

Responsibilities

• Design challenging, real-world problems to target core capability loss failures identified in a leading model.
• Content Generation: Integrate the problems into an Agentic development environment using Python, with instructions, environment, and metadata.
• A Golden solution that follows the instructions.
• Evaluation and Analysis: Evaluate cross-model performance on the tasks.
• Headroom Identification: Identify tasks where target model fails to pass all tests, classifying failures as logical reasoning.

Job description

AI Work Expert is seeking a contractor to design and develop real-world problems for an Agentic development environment. You will implement tasks in Python, create environments, and provide a golden solution and test notebooks to evaluate model performance.

The role emphasizes strong expertise in data science, ML, coding, or frontier STEM, with a demand for reliable weekly engagement (30+ hours). Remote work is supported, and the project is focused on advancing AI evaluation tasks.