Member of Technical Staff, Coding Research

SaidGig; New York (NY); Remote; USD 400,000 - 800,000; Full time

Job summary

SaidGig is seeking a results-driven researcher/engineer to advance frontier coding agents by designing benchmarks, data systems, and evaluation methods. This role blends AI research, software engineering, and model assessment to push next-generation coding models forward.

You will lead end-to-end evaluation initiatives, build scalable datasets and tooling, analyze model behavior, and collaborate with researchers and engineers to communicate findings to clients and stakeholders.

Qualifications

• Strong software engineering background with Python, C++, or comparable languages.
• Minimum 3 years of experience in software engineering, ML research, evaluation, or related technical discipline.
• Experience designing, reviewing, or validating benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, RL, model evaluation, or related AI systems.
• Ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills for investigating model behavior and deriving insights from complex systems.
• Excellent written and verbal communication, able to explain technical findings to varied audiences.
• Comfort working in a fast-moving research environment with ambiguity and evolving priorities.

Responsibilities

• Design and own evaluation frameworks for coding agents, including benchmark specifications, scoring methods, rubrics, and quality standards.
• Lead end-to-end research initiatives to measure and improve coding-model performance across a range of software engineering tasks.
• Develop high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Analyze model behavior and failure modes, identify weaknesses, and translate findings into training/evaluation improvements.
• Build tooling and infrastructure for large-scale experimentation, data generation, review workflows, and evaluation pipelines.
• Establish rigorous, reproducible best practices for coding-agent assessment and measurement quality.
• Partner with researchers, engineers, and applied AI teams to design experiments and assess emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research initiatives communicating performance and insights.

Skills

Python

C++

Software engineering

Machine learning

Evaluation methodologies

Analytical skills

Communication

Job description

Role Overview

Advance the evaluation and development of frontier coding agents by designing the benchmarks, methodologies, and data systems used to measure and improve next-generation coding models. This role combines AI research, software engineering, and model evaluation.

Key Responsibilities

• Design and own evaluation frameworks for coding agents, including benchmark specifications, scoring methods, rubrics, and quality standards.
• Lead end-to-end research initiatives to measure and improve coding-model performance across a range of software engineering tasks.
• Develop high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Analyze model behavior and failure modes, identify systematic weaknesses, and translate findings into improvements for training and evaluation.
• Build tooling and infrastructure for large-scale experimentation, data generation, review workflows, and evaluation pipelines.
• Establish rigorous, reproducible best practices for coding-agent assessment and measurement quality.
• Partner with researchers, engineers, and applied AI teams to design experiments and assess emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research initiatives that communicate model performance and insights.

Qualifications

• Strong software engineering background and expertise in Python, C++, or comparable programming languages.
• At least 3 years of experience in software engineering, machine learning, AI research, evaluation, or a related technical discipline.
• Experience designing, reviewing, or validating technical assessments, benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, reinforcement learning, model evaluation, or related AI systems.
• Ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills for investigating model behavior and deriving insights from complex technical systems.
• Excellent written and verbal communication skills, including the ability to explain technical findings to varied audiences.
• Comfort working in a fast-moving research environment with ambiguity and evolving priorities.

Preferred Qualifications

• Experience with frontier AI systems, coding agents, or model-evaluation research.
• Interest in how data, evaluations, and feedback mechanisms influence model capabilities.
• A record of independently driving ambiguous technical or research projects from concept through execution.
• Experience designing benchmarks or datasets for machine learning systems at scale.
• Familiarity with agentic workflows, tool use, reinforcement learning, or post-training methods.
• Publications, open-source contributions, or demonstrated technical leadership.

Work Terms

• Full-time, remote role.
• CompensationAnnual compensation range: $400, 000 to $800, 000.