Python Coding Annotator for AI Benchmarks (Remote)

TransPerfect; Paris; Hybride; EUR 24 000 - 42 000; Temps partiel

Résumé du poste

DataForce by TransPerfect is seeking a Coding Annotator with strong Python skills to support AI model evaluation. This part-time role (24 hours/week) is remote from France and focuses on creating coding prompts, benchmarks, and reference solutions for industry standards.

You will test model outputs, explore failures, and help refine RL workflows by building coding environments and performing validation checks. English proficiency is essential.

Qualifications

• Bachelor's degree in CS/Software Eng/Computer Eng or related field.
• 3+ years of work experience in related field.
• Excellent written and verbal English communication.

Responsabilités

• Create coding prompts and reference solutions aligned with industry benchmarks such as SWE-Bench and Terminal-Bench.
• Develop prompts focused on code refactoring, code generation, and problem-solving scenarios.
• Evaluate model outputs to identify errors, limitations, and failure patterns in reasoning, correctness, and execution.
• Design and maintain coding environments used for evaluation and RL pipelines.
• Execute coding-specific validation checks using established criteria and tools provided by other annotation teams.
• Document findings, evaluation results, and insights to support model improvement and training strategies.
• Perform detailed code reviews and annotations for accuracy and compliance.
• Collaborate with cross-functional teams to improve code quality and documentation.

Connaissances

Python

JavaScript

TypeScript

English proficiency

Code review

Attention to detail

Formation

Bachelor's degree in Computer Science/Software Engineering/Computer Engineering or related field

Outils

Java

Description du poste

DataForce by TransPerfect is seeking a Coding Annotator with strong Python skills to support AI model evaluation. This part-time role (24 hours/week) is remote from France and focuses on creating coding prompts, benchmarks, and reference solutions for industry standards.

You will test model outputs, explore failures, and help refine RL workflows by building coding environments and performing validation checks. English proficiency is essential.