Senior Software Engineer, AI Evaluation and Coding Agents

SaidGig; New York (NY); Remote; USD 138,000 - 207,000; Part time

Job summary

SaidGig seeks a remote independent-contractor to improve AI coding agents by reviewing work across real-world software repositories. You will assess task understanding, approach selection, and code correctness, providing precise feedback to strengthen model performance.

You will design rubrics, label data to high quality, build pipelines, and document findings for the team, collaborating with researchers and engineers to scale qualitative judgment into robust processes.

Qualifications

• 5+ years of hands-on software engineering experience.
• Proficiency in Python, TypeScript/JavaScript, Go, or another major production language.
• Experience working in substantial, real-world codebases.
• Strong code-review skills and technical judgment.
• Ability to clearly explain why an implementation is correct, incorrect, or could be improved.
• Strong written communication skills.
• Experience using modern LLMs or AI coding tools.
• Experience with LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training is helpful but not required.

Responsibilities

• Design, refine, and iterate on rubrics and evaluation criteria for preference data.
• Review and label data to a high quality standard, identifying subtle issues that may be missed by others.
• Build and maintain pipelines and infrastructure for data generation, collection, and evaluation workflows.
• Turn findings from data work into clear written updates, recommendations, and documentation for the team.
• Work closely with researchers and engineers to convert qualitative technical judgment into scalable processes.

Skills

Python

TypeScript

JavaScript

Go

Code review

Written communication

LLMs / AI coding tools

RLHF / data design

Job description

Role Overview

Help improve AI coding agents by reviewing their work across real-world software repositories. You will assess whether an agent understands a task, selects an appropriate approach, and produces code that is correct, robust, and maintainable. Your technical judgment and precise feedback will help create the evaluation signals used to strengthen model performance.

Key Responsibilities

• Design, refine, and iterate on rubrics and evaluation criteria for preference data.
• Review and label data to a high quality standard, identifying subtle issues that may be missed by others.
• Build and maintain pipelines and infrastructure for data generation, collection, and evaluation workflows.
• Turn findings from data work into clear written updates, recommendations, and documentation for the team.
• Work closely with researchers and engineers to convert qualitative technical judgment into scalable processes.

Qualifications

• At least 5 years of hands-on software engineering experience.
• Strong proficiency in Python, TypeScript or JavaScript, Go, or another major production language.
• Experience working in substantial, real-world codebases.
• Strong code-review skills and technical judgment.
• Ability to clearly explain why an implementation is correct, incorrect, or could be improved.
• Strong written communication skills.
• Experience using modern LLMs or AI coding tools.
• Experience with LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training is helpful but not required.

Work Terms

Remote independent-contractor engagement, available only to candidates located in North America.

Commitment of 20 to 40 hours per week, including at least 6 hours of overlap with Pacific Time each day.

Approximately 3 months in duration, with a start date as soon as possible.

Compensation $100 to $150 per hour.