Model Evaluator – AI/ML

Turing; United States; Remote; USD 3,300 - 5,000; Part time

Benefits offered by this job

Fully remote

Cutting-edge AI projects

Job summary

Turing is a fully remote AI company seeking experienced Software Engineers to evaluate AI coding solutions and improve AI-assisted software development. You will review AI agent runs, assess code quality, and provide structured ratings and evidence-backed feedback.

Candidates must have 3+ years of software engineering, strong knowledge of Python/JavaScript/Java/Go/PHP, and proficiency with Shell, Git, and Docker.

Qualifications

• Minimum 3+ years of overall software engineering experience.
• Strong experience with at least one of the listed languages (Python, JavaScript/TypeScript, Java, Go, PHP).
• Proficiency with Shell commands, Git, Docker, and basic software pipelines.
• Comfortable running and testing real-world projects locally.

Responsibilities

• Review AI coding agent runs and guidance documents for real software tasks.
• Rate each run on 10 quality areas and provide an overall score and ranking.
• Write concise rationales with evidence from the runs.
• Ensure compliance with the no-AI-rationales policy during evaluation.

Skills

Python

JavaScript/TypeScript

Java

Go

PHP

Tools

Shell

Git

Docker

CI/CD basics

Job description

Review and rate AI coding solutions for real software engineering tasks.

Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.

About the Role

We are seeking experienced Software Engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents. In this role, you will review and compare AI-generated coding solutions across real-world software engineering tasks. You will analyze complete agent execution workflows, evaluate code correctness, efficiency, safety, and overall quality, and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.

What does day-to-day look like:

• You’ll review three AI coding agents that were each given the same real developer request.
• You read each agent’s full run: the commands it ran, the files it changed, what it checked and its final message.
• Then you judge which run did the job best.

For every task you

• Rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.
• Give each run an overall score from 1 to 5 and rank the three.
• Write a short rationale explaining your ranking, with evidence from the runs.
• Required Skills:
• Minimum 3+ years of overall experience
• Strong experience with at least one of the following languages: Python, JavaScript/TypeScript, Java, Go, PHP or similar.
• Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.
• Comfortable running, modifying, and testing real-world projects locally.

Assessment & Selection Process

• Candidates must complete three practice evaluation tasks as part of the selection process.
• Each assessment requires quality ratings, overall scores, comparative rankings, and written rationales.
• A reviewer will assess each task for technical accuracy, consistency, and quality of reasoning.
• Candidates will be onboarded only after all three practice tasks have been approved.

IMPORTANT — STRICT NO-AI POLICY

• DO NOT USE CHATGPT OR ANY OTHER AI TOOL TO WRITE YOUR EVALUATION RATIONALES, EXPLANATIONS, OR REASONS.
• ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE.
• COMPLIANCE WILL BE CHECKED. ANY USE OF AI-GENERATED RATIONALES OR REASONS WILL RESULT IN AUTOMATIC REJECTION.

Perks of Freelancing With Turing

• Work in a fully remote environment.
• Opportunity to work on cutting-edge AI projects with leading LLM companies.

Offer Details

• Commitments Required: At least 8 hours per day and 40 hours per week with overlap of 4 hours with PST.
• Employment type : Contractor assignment (no medical/paid leave)
• Duration of contrac : 1 month; [expected start date is next week]
• Payment model: $25 per approved task
• Location: India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, LA

Evaluation Process (approximately 75 mins) :

• Assessment based review

Skills:

Machine Learning