AI Benchmark Architect: Craft Hard Problems to Stump Models

Lifted, an Upwork Company; Tallahassee (FL); On-site; USD 90,000 - 150,000; Full time

Job summary

Lifted, an Upwork Company, seeks a seasoned problem designer to build benchmarks of expert-level challenges that today’s best AI agents cannot solve. You will craft tasks from real-world expert domains and translate them into Linux-terminal challenges for evaluation.

The role focuses on designing robust benchmarks, ensuring models fail across multiple frontier AI systems, and delivering rigorous, publishable work. A strong technical background and proven research outputs are essential.

Qualifications

• Masters degree or higher in a technical field.
• At least 10 years of professional experience in a technical domain.
• At least one publication (academic or professional).

Responsibilities

• Design hard problems from real-world domains for AI benchmarking.
• Convert problems into Linux-terminal tasks that an AI must solve from scratch.
• Ensure a problem counts only if multiple frontier models fail on repeated trials.

Skills

Coding

Machine Learning

Systems

Security

Hardware

Education

Masters degree or higher

10+ years of professional experience in a technical domain

At least one publication

Job description

Lifted, an Upwork Company, seeks a seasoned problem designer to build benchmarks of expert-level challenges that today’s best AI agents cannot solve. You will craft tasks from real-world expert domains and translate them into Linux-terminal challenges for evaluation.

The role focuses on designing robust benchmarks, ensuring models fail across multiple frontier AI systems, and delivering rigorous, publishable work. A strong technical background and proven research outputs are essential.