Python Model Evaluator (AI Coding) · Turing $25/task remote in Asia/Africa/LATAM 2h ago

Benture; Pakistan; Remote; PKR 7,624,000 - 11,436,000; Part time

Job summary

Turing is hiring an experienced Python-focused software engineer for an AI training and evaluation initiative. You will review AI-generated coding solutions, analyze complete agent runs, and provide structured ratings and evidence-based feedback.

Responsibilities include ranking three agent runs, writing rationales, and ensuring rigorous evaluation across performance and safety criteria. Fully remote work with flexible collaboration for this AI project.

Qualifications

• Minimum 3+ years of software engineering experience.
• Strong experience with Python.
• Proficiency with shell commands, Git, Docker, and basic software pipeline setup.
• Comfortable running, modifying, and testing real-world projects locally.

Responsibilities

• Review three AI coding agents given the same developer task and compare results.
• Judge runs using 10 quality areas and assign an overall score.
• Provide a short rationale with evidence-based feedback.
• Rank the three runs and document the rationale.

Skills

Python

Shell commands

Git

Docker

Software pipeline setup

Tools

Git

Docker

Job description

Python Model Evaluator (AI Coding) | $25/task | Remote contractor

Turing, an AI company, is hiring experienced software engineers for an AI training and evaluation initiative focused on improving the coding capabilities of AI agents. You will review and compare AI-generated coding solutions across real-world software engineering tasks, analyzing complete agent execution workflows and providing structured ratings and evidence-based feedback.

What the day-to-day looks like

• Review three AI coding agents that were each given the same real developer request.
• Read each agent's full run: the commands it ran, the files it changed, what it checked, and its final message.
• Judge which run did the job best.
• For every task, rate each run on 10 quality areas, such as following instructions, whether the code works, efficiency, safety, honesty and communication.
• Give each run an overall score from 1 to 5 and rank the three.
• Write a short rationale explaining your ranking, with evidence from the runs.

Requirements

• Minimum 3+ years of overall software engineering experience.
• Strong experience with Python.
• Proficiency with shell commands, Git, Docker, and basic software pipeline setup.
• Comfortable running, modifying, and testing real-world projects locally.

Pay and schedule

• $25 per approved task.
• At least 8 hours per day and 40 hours per week, with 4 hours of overlap with PST.
• Contractor assignment for 1 month; expected start date is next week.
• No medical or paid leave.

Location

Fully remote. Eligible locations: India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, Mexico, LA.

Assessment and selection

• Complete three practice evaluation tasks as part of the selection process; each requires quality ratings, overall scores, comparative rankings, and written rationales.
• A reviewer assesses each task for technical accuracy, consistency, and quality of reasoning.
• Onboarding happens only after all three practice tasks are approved.
• The evaluation process takes approximately 75 minutes and is assessment-based.

Strict no-AI policy: Do not use ChatGPT or any other AI tool to write your evaluation rationales, explanations, or reasons. All reasoning must be written independently by the candidate. Compliance will be checked, and any use of AI-generated rationales or reasons results in automatic rejection.

The role offers a fully remote environment and the opportunity to work on AI projects with leading LLM companies.