Frontier AI Benchmark Architect for Reasoning Tasks

Turing; United States; Remote; USD 150,000 - 230,000; Full time

Job summary

Turing, based in San Francisco, seeks researchers to design and analyze benchmark tasks that push frontier AI reasoning. You will build Python-enabled environments, provide gold solutions, datasets, and tests, and rigorously evaluate cross-model performance to uncover core capability gaps.

The role emphasizes deep quantitative work in data science, ML, and coding, with flexibility to contribute 30+ hours weekly. Proficiency in Rust is a plus, and strong GitHub collaboration is essential.

Qualifications

• Expertise in data science, ML, coding, or frontier STEM with strong quantitative skills.
• Able to commit 30+ hours per week on weekdays.
• Rust proficiency is a plus.

Responsibilities

• Design challenging, real-world benchmark tasks to test a frontier model.
• Generate content and environment components using Python, including instructions, golden solution, datasets, libraries, and metadata.
• Prepare a test notebook with unit tests that solutions must pass.
• Evaluate cross-model performance on the designed tasks.
• Identify tasks where the target model fails due to logical reasoning.
• Analyze Agent Trajectory to extract core capability loss patterns.

Skills

GitHub contributor

Data science

Machine learning

Coding

Quantitative background

Rust

Job description

Turing, based in San Francisco, seeks researchers to design and analyze benchmark tasks that push frontier AI reasoning. You will build Python-enabled environments, provide gold solutions, datasets, and tests, and rigorously evaluate cross-model performance to uncover core capability gaps.

The role emphasizes deep quantitative work in data science, ML, and coding, with flexibility to contribute 30+ hours weekly. Proficiency in Rust is a plus, and strong GitHub collaboration is essential.