Remote Coding Research and Evaluation Scientist

SaidGig; New York (NY); Remote; USD 400,000 - 800,000; Full time

Job summary

SaidGig is seeking a results-driven researcher/engineer to advance frontier coding agents by designing benchmarks, data systems, and evaluation methods. This role blends AI research, software engineering, and model assessment to push next-generation coding models forward.

You will lead end-to-end evaluation initiatives, build scalable datasets and tooling, analyze model behavior, and collaborate with researchers and engineers to communicate findings to clients and stakeholders.

Qualifications

• Strong software engineering background with Python, C++, or comparable languages.
• Minimum 3 years of experience in software engineering, ML research, evaluation, or related technical discipline.
• Experience designing, reviewing, or validating benchmarks, coding tasks, or evaluation methodologies.
• Familiarity with large language models, coding agents, RL, model evaluation, or related AI systems.
• Ability to build tooling, automate workflows, and improve technical processes through systematic experimentation.
• Strong analytical skills for investigating model behavior and deriving insights from complex systems.
• Excellent written and verbal communication, able to explain technical findings to varied audiences.
• Comfort working in a fast-moving research environment with ambiguity and evolving priorities.

Responsibilities

• Design and own evaluation frameworks for coding agents, including benchmark specifications, scoring methods, rubrics, and quality standards.
• Lead end-to-end research initiatives to measure and improve coding-model performance across a range of software engineering tasks.
• Develop high-quality datasets, golden examples, and evaluation protocols for reliable assessment of frontier coding systems.
• Analyze model behavior and failure modes, identify weaknesses, and translate findings into training/evaluation improvements.
• Build tooling and infrastructure for large-scale experimentation, data generation, review workflows, and evaluation pipelines.
• Establish rigorous, reproducible best practices for coding-agent assessment and measurement quality.
• Partner with researchers, engineers, and applied AI teams to design experiments and assess emerging model capabilities.
• Contribute to technical reports, benchmark studies, and client-facing research initiatives communicating performance and insights.

Skills

Python

C++

Software engineering

Machine learning

Evaluation methodologies

Analytical skills

Communication

Job description

SaidGig is seeking a results-driven researcher/engineer to advance frontier coding agents by designing benchmarks, data systems, and evaluation methods. This role blends AI research, software engineering, and model assessment to push next-generation coding models forward.

You will lead end-to-end evaluation initiatives, build scalable datasets and tooling, analyze model behavior, and collaborate with researchers and engineers to communicate findings to clients and stakeholders.