Senior Software Engineer, AI Evaluation & Coding Agents

24 Mag; New York (NY); Remote; USD 138,000 - 207,000; Full time

Job summary

24-MAG LLC is seeking a Senior Software Engineer to remotely evaluate AI-generated coding solutions. You will review code across substantial real-world repositories, assess correctness and maintainability, and translate findings into structured evaluation signals.

The role focuses on rigorous assessment, rubric refinement, and collaboration with research and engineering teams. Remote NA engagement with flexible hours and milestone-driven work.

Qualifications

• Five years of hands-on software engineering experience.
• Proficiency in Python, TypeScript, JavaScript, Go, or Java.
• Strong code-review and technical-analysis skills.
• Experience with GitHub-based development workflows.
• Familiarity with CI/CD and production engineering practices.

Responsibilities

• AI-Generated Code Evaluation: review AI-produced code for correctness, robustness, and maintainability.
• Rubric & Preference Evaluation: design evaluation rubrics and assess agent performance.
• Repository & Engineering Analysis: analyze real-world codebases and broader architecture.
• Evaluation Infrastructure & Workflows: build and improve data generation and evaluation pipelines.
• Research & Technical Collaboration: summarize findings and provide actionable recommendations.
• Communicate complex technical issues clearly in structured language.

Skills

Python

TypeScript

JavaScript

Go

Java

Code review

Engineering analysis

GitHub workflows

CI/CD

Open-source familiarity

Tools

GitHub

CI/CD tools

Job description

24-MAG LLC is seeking a Senior Software Engineer to remotely evaluate AI-generated coding solutions. You will review code across substantial real-world repositories, assess correctness and maintainability, and translate findings into structured evaluation signals.

The role focuses on rigorous assessment, rubric refinement, and collaboration with research and engineering teams. Remote NA engagement with flexible hours and milestone-driven work.