Model Evaluator – DevOps

HumanitApp; Pakistan; Remote; PKR 7,862,000 - 11,197,000; Full time

Benefits offered by this job

Work in a fully remote environment

Cutting-edge AI projects with leadingL

Job summary

Turing is seeking experienced Software Engineers to support AI training and evaluation initiatives focused on coding capabilities. You will review and compare AI-generated solutions, analyze workflows, evaluate code, and provide structured feedback to improve AI-assisted software development.

Day-to-day involves reviewing runs, rating quality, and ranking three solutions. This contractor role requires at least 8 hours daily, 40 hours weekly, with a 4-hour overlap with PST, and offers remote work

Qualifications

• Minimum 3+ years of overall experience.
• Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.
• Comfortable running, modifying, and testing real‑world projects locally.

Responsibilities

• Review three AI coding agents' runs and judge which run did the job best.
• Rate runs on 10 quality areas and assign overall scores.
• Write a short rationale with evidence from the runs.
• Complete three practice evaluation tasks as part of the selection process.
• Ensure no AI-generated rationales; independent reasoning is required.

Skills

Shell commands

Git

CI/CD basics

Tools

Docker

Job description

Future projects, not immediate vacancies. Check each network's country and experience requirements.

Review and rate AI coding solutions for software engineering tasks.

About Turing

Turing is one of the world’s fastest-growing AI companies, accelerating the advancement and deployment of powerful AI systems. You’ll be at the forefront of evaluating how LLMs interact with real code, influencing the future of AI-assisted software development. This is a unique opportunity to blend practical software engineering with AI research.

About the Role

We are seeking experienced Software Engineers to support a frontier AI training and evaluation initiative focused on improving the coding capabilities of AI agents.

In this role, you will review and compare AI-generated coding solutions across real-world software engineering tasks. You will analyze complete agent execution workflows, evaluate code correctness, efficiency, safety, and overall quality, and provide structured ratings and evidence-based feedback to help improve AI-assisted software development.

What does day‑to‑day look like

You'll review three AI coding agents that were each given the same real developer request.

You read each agent's full run: the commands it ran, the files it changed, what it checked and its final message.

Then you judge which run did the job best.

Rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.

Give each run an overall score from 1 to 5 and rank the three.

Write a short rationale explaining your ranking, with evidence from the runs.

Required Skills

• Minimum 3+ years of overall experience
• Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.
• Comfortable running, modifying, and testing real‑world projects locally.

Assessment & Selection Process

Candidates must complete three practice evaluation tasks as part of the selection process.

Each assessment requires quality ratings, overall scores, comparative rankings, and written rationales.

A reviewer will assess each task for technical accuracy, consistency, and quality of reasoning.

Candidates will be onboarded only after all three practice tasks have been approved.

IMPORTANT

STRICT NO-AI POLICY

DO NOT USE CHATGPT OR ANY OTHER AI TOOL TO WRITE YOUR EVALUATION RATIONALES, EXPLANATIONS, OR REASONS.

ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE.

COMPLIANCE WILL BE CHECKED. ANY USE OF AI-GENERATED RATIONALES OR REASONS WILL RESULT IN AUTOMATIC REJECTION.

Perks of Freelancing With Turing

• Work in a fully remote environment.
• Opportunity to work on cutting‑edge AI projects with leading LLM companies.

Commitments Required: At least 8 hours per day and 40 hours per week with overlap of 4 hours with PST.

Employment type : Contractor assignment (no medical/paid leave)

Duration of contrac : 1 month; [expected start date is next week]

Payment model: $25 per approved task

Evaluation Process (approximately 75 mins) :

Assessment based review

DevOps

Compensation context

The listing states $25 per approved task. Confirm the final rate, workload, and payment terms during the official application process.

• Proficiency with Shell commands, Git, Docker, and basic software pipeline setup.
• ALL RATIONALES AND REASONING MUST BE WRITTEN INDEPENDENTLY BY THE CANDIDATE.

Remote Country eligibility: check official listing Work type not specified