Remote Python Engineer for AI Benchmarking

24-Mag Llc; New York (NY); Remote; USD 75,768 - 117,096; Full time

Job summary

24-MAG LLC is offering a full-time remote consulting opportunity for experienced software engineers with strong Python development, debugging, and AI-assisted coding experience. You will design, implement, and review multi-step tasks to evaluate frontier AI models in realistic development workflows.

The role emphasizes reference solution development, environment setup, testing, and thorough documentation, with approximately 35 hours per week and competitive hourly rates in the United States.

Qualifications

• Strong hands-on Python development, debugging, and scripting.
• Experience with Git and standard software development workflows.
• Ability to design, implement, and review multi-step coding tasks.
• Excellent technical writing and documentation skills.

Responsibilities

• Design realistic, multi-step Python software engineering challenges.
• Develop reference solutions with setup, tests, and validation.
• Evaluate AI coding agents and identify failure patterns.
• Perform peer reviews to improve task quality and fairness.
• Collaborate with researchers and other task authors.

Skills

Python

Debugging

Version control

Technical writing

AI-assisted coding

Education

MSc or PhD in CS/Software Eng

Tools

CI/CD pipelines

Containers

Job description

24-MAG LLC is offering a full-time remote consulting opportunity for experienced software engineers with strong Python development, debugging, and AI-assisted coding experience. You will design, implement, and review multi-step tasks to evaluate frontier AI models in realistic development workflows.

The role emphasizes reference solution development, environment setup, testing, and thorough documentation, with approximately 35 hours per week and competitive hourly rates in the United States.