AI Benchmark Engineer (Python) — Remote

Weekday AI; United States; Remote; USD 83,000 - 124,000; Part time

Job summary

Weekday AI is seeking experienced Software Engineers to design and validate multi-step benchmarks that mirror real-world software tasks for frontier AI models. You will implement complex tasks, configure environments, and reason through solutions alongside AI researchers.

This is a fully remote, part-time engagement requiring about 35 hours per week, with weekly payments and strong emphasis on code quality, reproducibility, and rigorous validation checks.

Qualifications

• Master's degree, PhD, or equivalent practical experience in Computer Science, Software Engineering, or another STEM discipline involving substantial programming.
• Minimum 1 year of professional software engineering or research engineering experience.
• Strong hands-on Python including design, scripting, debugging, testing and clean code.
• Solid experience with Git, IDEs and modern software development workflows.
• Experience with AI coding assistants, prompt engineering, or AI agent workflows is preferred.

Responsibilities

• Design realistic, multi-step software engineering challenges based on practical workflows.
• Develop reference solutions in Python that are reproducible and validated.
• Leverage AI coding assistants while evaluating their strengths and failure modes on tasks.
• Review tasks created by peers, providing feedback on correctness and difficulty.
• Analyze AI-generated solutions to identify errors and improvement opportunities.
• Collaborate with AI researchers and engineers to refine evaluation methodologies.

Skills

Python

Git

Software engineering

Debugging

Testing

Code review

Documentation

Education

Master's/PhD or equivalent

Tools

Git

IDEs

Job description

Weekday AI is seeking experienced Software Engineers to design and validate multi-step benchmarks that mirror real-world software tasks for frontier AI models. You will implement complex tasks, configure environments, and reason through solutions alongside AI researchers.

This is a fully remote, part-time engagement requiring about 35 hours per week, with weekly payments and strong emphasis on code quality, reproducibility, and rigorous validation checks.