Remote AI Code Evaluation Engineer — Benchmark & Review

SaidGig; New York (NY); Remote; USD 55,000 - 96,000; Part time

Benefits offered by this job

Fully remote work

Job summary

SaidGig seeks a skilled software engineer to evaluate frontier AI models by reviewing AI-generated code against real-world tasks and shaping evaluation datasets. You will diagnose correctness, performance, and reliability, enabling high-quality benchmarks and grading rubrics.

Ideal candidates have 3+ years in software engineering, strong Python or other language skills, and familiarity with AI data annotation and model evaluation.

Qualifications

• Bachelor's or Master’s degree in Computer Science, Software Engineering, or a related technical field.
• Minimum 3 years of professional software engineering experience.
• Strong proficiency in Python, Java, C/C++, Go, Swift, Objective-C, PHP, or SQL.
• Solid understanding of data structures, algorithms, software design principles, and debugging methodologies.
• Experience performing code reviews and evaluating code quality in production or large-scale codebases.
• Ability to analyze complex technical problems with minimal supervision.
• Familiarity with Git and modern software development workflows.
• Strong written communication skills and attention to detail.
• Experience with AI/ML data annotation, NLP, prompt engineering, model evaluation, or LLM-related projects is a plus.

Responsibilities

• Review AI-generated code for correctness, efficiency, maintainability, and compliance with task requirements.
• Analyze software engineering tasks and validate whether proposed solutions meet expected outcomes.
• Debug code, reproduce issues, and verify fixes across multiple programming environments.
• Evaluate model-generated explanations and reasoning for technical accuracy and soundness.
• Create, refine, and maintain evaluation datasets, benchmarks, and grading rubrics for coding tasks.
• Identify edge cases, failure modes, and areas where models struggle with software engineering problems.
• Document findings clearly and provide structured feedback to improve evaluation consistency and quality.
• Collaborate with project teams to establish and maintain quality standards and evaluation methodologies.

Skills

Python

Code review

Debugging

Software engineering

AI/ML evaluation

Documentation

Education

Bachelor's or Master's in CS/Software Eng

Tools

Git

Docker

CI/CD

Job description

SaidGig seeks a skilled software engineer to evaluate frontier AI models by reviewing AI-generated code against real-world tasks and shaping evaluation datasets. You will diagnose correctness, performance, and reliability, enabling high-quality benchmarks and grading rubrics.

Ideal candidates have 3+ years in software engineering, strong Python or other language skills, and familiarity with AI data annotation and model evaluation.