Model Evaluator – AI/ML

HumanitApp; Pakistan; Remote; PKR 7,624,000 - 11,436,000; Full time

Benefits offered by this job

Fully remote

Job summary

Turing is seeking experienced Software Engineers to support AI training and evaluation, focusing on reviewing AI-generated coding solutions for real software tasks. You will assess agent runs, measure correctness and efficiency, and provide evidence-based feedback to improve AI-assisted software development.

The role is fully remote, contractor-based, with a pay- per-task model ($25 per approved task) and a defined 1-month engagement.

Qualifications

• Minimum 3+ years of overall experience.
• Proficiency with shell commands, Git and Docker.
• Comfortable running and testing real-world projects locally.

Responsibilities

• Review and rate AI coding solutions across real software tasks.
• Evaluate code correctness, efficiency, safety, and quality.
• Provide structured ratings and evidence-based feedback to improve AI-assisted software development.
• Rank three agent runs and write concise rationales.

Skills

Shell commands

Git

3+ years experience

Basic software pipelines

Tools

Docker

Job description

Future projects, not immediate vacancies. Check each network's country and experience requirements.

Review and rate AI coding solutions for real software engineering tasks.

About Turing

Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.

About the Role

We are seeking experienced Software Engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents.

In this role, you will review and compare AI-generated coding solutions across real-world software engineering tasks. You will analyze complete agent execution workflows, evaluate code correctness, efficiency, safety, and overall quality, and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.

What does day-to-day look like

You’ll review three AI coding agents that were each given the same real developer request.

You read each agent’s full run: the commands it ran, the files it changed, what it checked and its final message.

Then you judge which run did the job best.

For every task you

Rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.

Give each run an overall score from 1 to 5 and rank the three.

Write a short rationale explaining your ranking, with evidence from the runs.

Required Skills

Minimum 3+ years of overall experience

Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.

Comfortable running, modifying, and testing real-world projects locally.

Assessment & Selection Process

Candidates must complete three practice evaluation tasks as part of the selection process.

Each assessment requires quality ratings, overall scores, comparative rankings, and written rationales.

A reviewer will assess each task for technical accuracy, consistency, and quality of reasoning.

Candidates will be onboarded only after all three practice tasks have been approved.

IMPORTANT

STRICT NO-AI POLICY

DO NOT USE CHATGPT OR ANY OTHER AI TOOL TO WRITE YOUR EVALUATION RATIONALES, EXPLANATIONS, OR REASONS.

ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE.

COMPLIANCE WILL BE CHECKED. ANY USE OF AI-GENERATED RATIONALES OR REASONS WILL RESULT IN AUTOMATIC REJECTION.

Perks of Freelancing With Turing

Work in a fully remote environment.

Opportunity to work on cutting-edge AI projects with leading LLM companies.

Commitments Required: At least 8 hours per day and 40 hours per week with overlap of 4 hours with PST.

Employment type : Contractor assignment (no medical/paid leave)

Duration of contrac : 1 month; [expected start date is next week]

Payment model: $25 per approved task

Evaluation Process (approximately 75 mins)

Assessment based review

Source: Turing official job description, checked 2026-11-09. Turing decides selection and final terms. Referral rewards are separate from applicant compensation.

Compensation context

The listing states $25 per approved task. Confirm the final rate, workload, and payment terms during the official application process.

• Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.
• ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE.

Remote · Country eligibility: check the official listing · Work type not specified

Task-based pay; confirm amount per accepted task

Global, fully remote · Work type not specified

Task-based pay; confirm amount per accepted task