Remote Python Engineer (AI Benchmarking & Task Design)

24-MAG; New York (NY); Remote; USD 76,000 - 117,000; Full time

Benefits offered by this job

Fully remote

Job summary

24-MAG LLC is offering a fully remote, full-time consulting opportunity for experienced software engineers with strong Python skills. You will design realistic, multi-step coding tasks and build verifiable Python reference solutions, including environment setup and tests.

The role evaluates AI coding assistants and requires rigorous documentation and peer-review contributions. The ideal candidate has 1+ year in software or research engineering, strong Python, and fluent Git/IDE workflows.

Qualifications

• At least 1 year of software engineering or research engineering experience.
• Strong hands-on Python scripting, implementation, and debugging skills.
• Fluent with Git, IDEs, repositories, and standard software development workflows.
• Ability to configure environments, dependencies, tooling, and validation processes.

Responsibilities

• Create realistic, multi-step software engineering challenges based on practical development workflows.
• Design technically demanding problems requiring implementation, debugging, environment configuration, and analytical reasoning.
• Define clear requirements, constraints, expected outputs, and acceptance criteria.
• Ensure tasks assess genuine software engineering capability rather than superficial code generation.
• Build complete and verifiable reference solutions in Python with setup, tests, and validation checks.
• Document implementation decisions and expected behavior clearly.
• Evaluate how frontier models approach complex coding and debugging tasks using AI assistants.
• Identify errors, unsupported assumptions, inefficient approaches, and incomplete solutions.
• Document failure patterns and provide evidence for evaluation conclusions.
• Review tasks and reference solutions created by others; provide actionable feedback.

Job description

24-MAG LLC is offering a fully remote, full-time consulting opportunity for experienced software engineers with strong Python skills. You will design realistic, multi-step coding tasks and build verifiable Python reference solutions, including environment setup and tests.

The role evaluates AI coding assistants and requires rigorous documentation and peer-review contributions. The ideal candidate has 1+ year in software or research engineering, strong Python, and fluent Git/IDE workflows.