Top GitHub Contributors – Agentic Review Project

Turing; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

Turing, based in San Francisco, seeks researchers to design and analyze benchmark tasks that push frontier AI reasoning. You will build Python-enabled environments, provide gold solutions, datasets, and tests, and rigorously evaluate cross-model performance to uncover core capability gaps.

The role emphasizes deep quantitative work in data science, ML, and coding, with flexibility to contribute 30+ hours weekly. Proficiency in Rust is a plus, and strong GitHub collaboration is essential.

Qualifications

• Expertise in data science, ML, coding, or frontier STEM with strong quantitative skills.
• Able to commit 30+ hours per week on weekdays.
• Rust proficiency is a plus.

Responsibilities

• Design challenging, real-world benchmark tasks to test a frontier model.
• Generate content and environment components using Python, including instructions, golden solution, datasets, libraries, and metadata.
• Prepare a test notebook with unit tests that solutions must pass.
• Evaluate cross-model performance on the designed tasks.
• Identify tasks where the target model fails due to logical reasoning.
• Analyze Agent Trajectory to extract core capability loss patterns.

Skills

GitHub contributor

Data science

Machine learning

Coding

Quantitative background

Rust

Job description

Design and analyze challenging benchmark tasks to enhance AI reasoning.

About Us

Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. Turing supports customers in two ways: first, by accelerating frontier research with high-quality data, advanced training pipelines, plus top AI researchers who specialize in software engineering, logical reasoning, STEM, multilinguality, multimodality, and agents; and second, by applying that expertise to help enterprises transform AI from proof of concept into proprietary intelligence with systems that perform reliably, deliver measurable impact, and drive lasting results on the P&L.

Project Overview

We are seeking high-caliber researchers and technical experts. The core objective of this role is to enhance the reasoning and problem-solving capabilities of a target frontier model by designing, validating, and analyzing challenging benchmark tasks.

What the role entails:

• Task Design and Development: Design challenging, real-world problems that serve as the foundation for a model. Problems should be constructed to target specific core capability loss failures identified in a leading model.
• Content Generation: Integrate the problems into an Agentic development environment, preparing all necessary components using Python, which include:
  • Detailed Instructions and an overview of the required task.
  • A Golden solution that follows the instructions.
  • The necessary Environment, including datasets, Python libraries, and metadata.
  • A Test notebook containing unit tests that solutions must pass.
• Evaluation and Analysis: Evaluate the cross model’s performance on the tasks.
• Headroom Identification: Identify tasks where target model fails to pass all tests, specifically classifying the failure as a logical reasoning failure.
• Loss Extraction: Analyze the agent’s steps (Agent Trajectory) to observe and extract core capability loss patterns from the model.

Required Skills

• Highly-skilled GitHub contributors.
• Applicants must have strong expertise in either data science, ML, coding, or a deep quantitative background in frontier STEM.
• Able to engage reliably for 30+ hrs per week (on weekdays).

Evaluation Process

• The application process takes ~15 minutes.
• Completion of an AI video interview is required.

Skills:

Rust