AI Benchmark Architect (Remote) | Python

Weekday 1; United States; Remote; USD 83,000 - 124,000; Full time

Benefits offered by this job

Fully remote

Weekly payments

Job summary

Weekday 1 is seeking experienced Software Engineers for a fully remote, contractor engagement to build multi-step software engineering benchmarks for frontier AI models. You will implement, debug, configure environments, and reason about tasks alongside AI researchers.

The role emphasizes Python reference solutions, reproducibility, and rigorous validation checks, with approximately 35 hours per week. Weekly payments apply for this remote engagement.

Qualifications

• Master's degree, PhD, or equivalent practical experience in CS/Software Engineering or STEM
• Minimum 1 year of professional software engineering or research engineering experience
• Strong Python proficiency including design, scripting, debugging, testing, and clean code
• Solid experience with Git, IDEs, and modern software development workflows
• Good understanding of software architecture, debugging methods, and best practices
• Experience with AI coding assistants, prompt engineering, or AI agent workflows is preferred

Responsibilities

• Design realistic, multi-step software engineering benchmarks reflecting typical development workflows
• Develop comprehensive Python reference solutions that are reproducible and validated
• Use AI coding assistants as part of the workflow while evaluating their strengths and failure modes
• Review benchmark tasks for technical correctness, clarity, completeness, and difficulty
• Analyze AI-generated solutions to identify errors, debugging challenges, and reasoning gaps
• Collaborate with AI researchers and engineering teams to refine evaluation methodologies

Skills

Python

Git

Debugging

Software design

Code testing

Education

Master's degree / PhD in CS or STEM

Tools

IDEs

AI coding assistants

Job description

Weekday 1 is seeking experienced Software Engineers for a fully remote, contractor engagement to build multi-step software engineering benchmarks for frontier AI models. You will implement, debug, configure environments, and reason about tasks alongside AI researchers.

The role emphasizes Python reference solutions, reproducibility, and rigorous validation checks, with approximately 35 hours per week. Weekly payments apply for this remote engagement.