Senior Software Engineer, LLM Evaluation

SaidGig; New York (NY); Remote; USD 120,000 - 180,000; Part time

Job summary

SaidGig is seeking a software engineer to help advance AI language models by creating high-quality datasets, benchmarks, and evaluations. You will work across Python, JavaScript (ReactJS), C/C++, Java, Rust, and Go to develop reference solutions and assess AI-generated output for engineering quality.

You will curate code examples, refine AI-driven coding solutions, and partner with researchers and product teams to ensure performance against industry benchmarks.

Qualifications

• Several years of software engineering experience at a top-tier product company.
• Strong full-stack development ability and ability to deploy production-grade software.
• Deep knowledge of architecture, design, debugging, and code quality.
• Excellent written and verbal communication with structured evaluation rationales.

Responsibilities

• Curate code examples, build solutions, and correct code across multiple languages for AI model training initiatives.
• Evaluate and refine AI-generated code for efficiency, scalability, and reliability.
• Partner with cross-functional teams to improve AI-driven coding solutions against benchmarks.
• Build agents that verify code quality and identify recurring error patterns.
• Develop and assess model capabilities across the software lifecycle, including prototyping, architecture design, API design, production implementation, launch, experiments, monitoring, operations, and maintenance.
• Design automated verification mechanisms for solutions to software engineering tasks.

Skills

Full-stack development

Production-grade software

Code review

Communication

Experience with AI datasets

Tools

Python

JavaScript/ReactJS

C/C++

Java

Rust

Go

Job description

Role Overview

Help advance large language models by creating high-quality software engineering datasets, benchmarks, and evaluations alongside AI researchers. You will work across Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go to develop reference solutions, improve code samples, and assess AI-generated output for real-world engineering quality.

Key Responsibilities

• Curate code examples, build solutions, and correct code across Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go for AI model training initiatives.
• Evaluate and refine AI-generated code for efficiency, scalability, and reliability.
• Partner with cross-functional teams to improve AI-driven coding solutions against industry performance benchmarks.
• Build agents that verify code quality and identify recurring error patterns.
• Develop and assess model capabilities across the software engineering lifecycle, including prototyping, architecture design, API design, production implementation, launch, experiments, monitoring, operations, and maintenance.
• Design automated verification mechanisms for solutions to software engineering tasks.

Qualifications

• Several years of software engineering experience, including at least 2 years of continuous full-time experience at a top-tier product company.
• Strong full-stack application development experience and the ability to deploy scalable, production-grade software with modern languages and tools.
• Deep knowledge of software architecture, design, development, debugging, and code-quality and code-review assessment.
• Excellent written and verbal communication skills, with the ability to provide clear, structured evaluation rationales.

Work Terms

Remote contract engagement.

Flexible schedule of at least 10 hours per week, with up to 40 hours per week available.

Some overlap with Pacific Time is required.

Independent contractor role, with no medical benefits or paid leave.