Software Engineer – AI Code Evaluation & Benchmarking (US candidates only)

Turing

United States

Remote

USD 83,000 - 152,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Fully remote
Cutting-edge AI projects

Job summary

Turing is seeking experienced Software Engineers to evaluate and improve AI coding systems. You will review AI-generated code, validate solutions, and help build evaluation datasets and benchmarks. This remote freelance role emphasizes code review, debugging, and applying software engineering judgment to real-world tasks.

The position focuses on measuring and improving coding performance in frontier AI models, with collaboration across project teams and the creation of robust grading rubrics.

Qualifications

  • Bachelor's or Master's degree in CS, Software Engineering, or a related field.
  • 3+ years of professional software engineering experience.
  • Strong proficiency in Python, Java, C/C++, Go, Swift, Objective-C, PHP or SQL.
  • Strong understanding of data structures, algorithms, and debugging.
  • Experience performing code reviews and evaluating code quality in large codebases.
  • Familiarity with Git and modern development workflows.
  • Experience with AI/ML data annotation, NLP, or model evaluation is a plus.

Responsibilities

  • Review and evaluate AI-generated code for correctness, efficiency, and maintainability.
  • Analyze software tasks and validate solutions meet expected outcomes.
  • Debug code, reproduce issues, and verify fixes across environments.
  • Assess model explanations and implementations for technical accuracy.
  • Create, refine, and maintain evaluation datasets, benchmarks, and rubrics.
  • Identify edge cases and failure modes where AI struggles with software tasks.
  • Document findings clearly and provide structured feedback to improve quality.
  • Collaborate with teams to set quality standards and evaluation methodologies.

Skills

Python
Java
C/C++
Go
Swift
SQL
Data structures
Algorithms
Code review
Debugging
Git

Education

Bachelor's or Master's in CS/Software Eng

Tools

Docker
Git

Job description

About Turing

Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. Turing helps leading AI labs improve the reasoning, problem-solving, and decision-making capabilities of large language models (LLMs) through high-quality human feedback, evaluation, and training data.

Role Overview

We are looking for experienced Software Engineers to help evaluate, benchmark, and improve the coding capabilities of frontier AI models. In this role, you will assess AI-generated code, validate solutions against real-world software engineering tasks, identify correctness and quality issues, and contribute to the development of high-quality evaluation datasets and benchmarks.

This position is ideal for engineers who enjoy code review, debugging, problem-solving, and applying strong software engineering judgment to complex technical scenarios. Your work will directly contribute to measuring and improving the performance of advanced AI coding systems.

What Does Day-to-Day Look Like?
  • Review and evaluate AI-generated code for correctness, efficiency, maintainability, and adherence to requirements.
  • Analyze software engineering tasks and validate whether proposed solutions meet expected outcomes.
  • Debug code, reproduce issues, and verify fixes across different programming environments.
  • Assess model-generated explanations, reasoning, and implementation approaches for technical accuracy.
  • Create, refine, and maintain evaluation datasets, benchmarks, and grading rubrics for coding tasks.
  • Identify edge cases, failure modes, and areas where AI systems struggle with software engineering problems.
  • Document findings clearly and provide structured feedback to improve evaluation quality and consistency.
  • Collaborate with project teams to establish quality standards and evaluation methodologies.
Requirements
  • Bachelor's or Master's degree in Computer Science, Software Engineering, or a related technical field.
  • 3+ years of professional software engineering experience.
  • Strong proficiency in one or more of the following languages: Python, Java, C/C++, Go, Swift, Objective-C, PHP, or SQL.
  • Strong understanding of data structures, algorithms, software design principles, and debugging methodologies.
  • Experience performing code reviews and evaluating code quality in production or large-scale codebases.
  • Ability to analyze complex technical problems and assess solution correctness with minimal supervision.
  • Familiarity with version control systems (e.g., Git) and modern software development workflows.
  • Strong written communication skills and attention to detail.
  • Experience with AI/ML data annotation, NLP, prompt engineering, model evaluation, or LLM-related projects is a plus.
  • Experience evaluating AI-generated code, benchmark creation, or software quality assessment is highly preferred.
Perks of Freelancing With Turing
  • Work in a fully remote environment.
  • Opportunity to work on cutting-edge AI projects with leading LLM companies.
Offer Details
  • Commitments Required: At least 4 hours per day and minimum 20 hours per week with overlap of 4 hours with PST.
  • Engagement type : Contractor assignment (no medical/paid leave)
  • Duration of contract : 1 month; [expected start date is next week]
  • Location: US only
Evaluation Process
  1. Online automated coding challenge for Python and Docker test (RHLF)
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Scientist/Analyst
Data Scientist/Analyst

Turing • United States

Remote
USD 83,000 - 124,000
Fully remote work environment
Cutting-edge AI projects with leadingL
Model Evaluator – AI/ML
Model Evaluator – AI/ML

Turing • United States

Remote
USD 3,300 - 5,000
Fully remote
Cutting-edge AI projects
Senior Software Engineer - 35501
Senior Software Engineer - 35501

Turing • San Francisco (CA)

Remote
USD 68,880 - 206,640
Software Engineer – AI Research & Evaluation (US-based)
Software Engineer – AI Research & Evaluation (US-based)

LLM Trainers • San Francisco (CA)

Hybrid
USD 96,000 - 165,000
Senior Software Engineer - 35501
Senior Software Engineer - 35501

Turing • New York (NY)

Remote
USD 68,880 - 206,640
Remote Software Developer
Remote Software Developer

Turing • Dallas (TX)

Remote
USD 55,000 - 96,000
Remote work
Flexible schedule
Competitive hourly rate
Remote Software Developer
Remote Software Developer

Turing • Texas

Remote
USD 83,000 - 152,000
Senior Python Engineer – LLM Evaluation (US-based)
Senior Python Engineer – LLM Evaluation (US-based)

Aitrainergigs • United States

On-site
USD 83,000 - 138,000
Remote Full Stack Developer
Remote Full Stack Developer

Turing • Pittsburgh

Remote
USD 83,000 - 138,000
Senior Software Engineer – LLM Evaluation
Senior Software Engineer – LLM Evaluation

HumanitApp • California

On-site
USD 83,000 - 124,000