Senior Software Engineer, LLM Evaluation (US/Canada/WEU)

SaidGig; Little Canada (MN); Remote; USD 83.000 - 165.000; Teilzeit

Zusammenfassung

SaidGig is seeking a contractor to design and evaluate high-quality datasets and evaluations that advance large language models for code. You will curate code examples in multiple languages, assess AI-generated implementations for production readiness, and help build verification agents across the software lifecycle.

You will work with researchers and engineering teams to ensure benchmarks reflect industry performance and drive improvements in model capabilities and code reliability.

Qualifikationen

• At least 3 years of professional software engineering experience.
• Experience building full-stack applications and deploying production-grade software.
• Deep knowledge of software architecture, design, development practices, debugging, and code quality.
• Strong written and oral communication skills for evaluation rationales and documentation.

Aufgaben

• Curate and author code examples, solutions, and corrections for model training and benchmarking.
• Evaluate AI-generated code for correctness, efficiency, scalability, and reliability with structured feedback.
• Collaborate with research and engineering teams to align datasets with benchmarks.
• Design and build automated verification agents to validate software tasks and detect error patterns.
• Formulate and test hypotheses about model performance across the software engineering lifecycle.

Kenntnisse

Python

JavaScript / ReactJS

C / C++

Java

Rust

Go

Full-stack development

Code quality / review

Strong communication

Jobbeschreibung

Role Overview

Design and evaluate high-quality datasets and evaluations that advance large language models for code. You will work with researchers to curate code examples, produce precise solutions and corrections across multiple languages, assess AI-generated implementations for production readiness, and help build verification agents that detect errors and measure model capabilities across the software engineering lifecycle.

Key Responsibilities

• Curate and author code examples, solutions, and corrections in Python, JavaScript including ReactJS, C and C++, Java, Rust, and Go for model training and benchmarking.
• Evaluate AI-generated code for correctness, efficiency, scalability, and reliability, and provide structured feedback and improvements.
• Collaborate with cross-functional research and engineering teams to align datasets and evaluations with industry performance benchmarks.
• Design and build agents or automated verification mechanisms that can validate solutions to software engineering tasks and identify recurring error patterns.
• Formulate and test hypotheses about model performance on steps of the software engineering cycle, including prototyping, architecture and API design, production implementation, launch, experimentation, monitoring, and operational maintenance.

Qualifications

• At least 3 years of professional software engineering experience.
• Proven experience building full-stack applications and deploying scalable, production-grade software using modern languages and tooling.
• Deep knowledge of software architecture, design, development practices, debugging, and code quality or review assessment.
• Strong written and oral communication skills, able to produce clear, structured evaluation rationales and code documentation.

Work Terms

Commitment: flexible engagement, minimum 10 hours per week, up to 40 hours per week. Engagement type: Contractor, no medical or paid leave provided. Duration: 1 month initial term, with potential extension based on performance and fit. Location: Remote, candidates must be based in the United States, Canada, or Western Europe countries such as Austria, Belgium, France, Germany, and similar WEU locations.

Eligibility

Applicants must be legally located in the United States, Canada, or a Western European country as listed above. This role is offered to independent contractors only, and does not include employee benefits.