AI Coding Agent Evaluation Engineer

Joblet-AI; United States; A distancia; GBP 149.000 - 194.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Equity compensation

Health insurance

401(k) with match

Descripción de la vacante

micro1 is seeking a Member of Technical Staff to advance evaluation and development of frontier coding agents from a remote setting. You will design benchmarks, evaluation methodologies, and data systems shaping how next-generation coding models are measured and improved.

You will lead end-to-end research on measuring and enhancing coding model performance across software engineering tasks, develop high-quality datasets and protocols, and build tooling for large-scale experimentation and review

Formación

• Strong software engineering background with Python, C++, or comparable languages.
• 3+ years of experience in software engineering, ML, AI research, evaluation, or related disciplines.
• Experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, reinforcement learning, or related AI systems.
• Proven ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills to investigate model behavior and derive insights from complex systems.
• Excellent written and verbal communication skills for diverse audiences.
• Able to work in fast-moving research environments with ambiguity and evolving priorities.

Responsabilidades

• Design and own evaluation frameworks for coding agents, including benchmark specs, scoring methodologies, rubrics, and quality standards.
• Lead end-to-end research initiatives to measure and improve coding model performance across diverse software tasks.
• Develop high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Analyze model behavior and failure modes; identify weaknesses and translate findings into improvements for training and evaluation.
• Build tooling and infrastructure to support large-scale experimentation, data generation, and evaluation pipelines.
• Establish best practices for coding-agent assessment, ensuring rigor, reproducibility, and measurement quality.
• Partner with researchers and engineers to design experiments and evaluate emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research initiatives.

Conocimientos

Python

C++

Machine learning

Evaluation frameworks

Software engineering

Communication

Descripción del empleo

micro1 is seeking a Member of Technical Staff to advance evaluation and development of frontier coding agents from a remote setting. You will design benchmarks, evaluation methodologies, and data systems shaping how next-generation coding models are measured and improved.

You will lead end-to-end research on measuring and enhancing coding model performance across software engineering tasks, develop high-quality datasets and protocols, and build tooling for large-scale experimentation and review