Senior ML Research Engineer - Coding-Agent Benchmarking

24-MAG; United States; Remote; USD 600,000 - 1,300,000; Full time

Job summary

24-MAG LLC. seeks experienced technical professionals to advance evaluation and development of frontier coding agents.

The role sits at the intersection of AI research, software engineering, and model evaluation, designing benchmarks, datasets, and technical systems that gauge and improve coding models. The ideal candidate has a strong software-engineering background, 3+ years in ML/AI research or evaluation, and experience crafting benchmarks and evaluation methodologies.

Qualifications

• Strong software-engineering background with Python/C++.
• At least 3 years of experience in ML/AI research or evaluation.
• Experience designing/validating benchmarks and assessment methodologies.

Responsibilities

• Design and own evaluation frameworks for advanced coding agents.
• Develop benchmark specifications, scoring rubrics, and quality standards.
• Measure coding-model performance across diverse software tasks.
• Analyze incorrect reasoning, tool-use failures, and execution gaps.
• Build research tooling and collaborate with researchers and engineers.

Skills

Python

C++

Software engineering

Machine learning

Model evaluation

Benchmark design

Automation & tooling

Communication

Analytical skills

Research adaptability

Tools

Research tooling

Experimentation pipelines

Job description

24-MAG LLC. seeks experienced technical professionals to advance evaluation and development of frontier coding agents.

The role sits at the intersection of AI research, software engineering, and model evaluation, designing benchmarks, datasets, and technical systems that gauge and improve coding models. The ideal candidate has a strong software-engineering background, 3+ years in ML/AI research or evaluation, and experience crafting benchmarks and evaluation methodologies.