Python Coding Annotator for AI Benchmarks (Paris, Part-Time)

TransPerfect; Paris; Hybride; EUR 25 000 - 42 000; Temps partiel

Résumé du poste

DataForce by TransPerfect in Paris, France is seeking Software Engineers to join as Coding Annotators to support AI model evaluation. You will create high-quality coding prompts and reference solutions, benchmark model performance, and identify failure cases across internal and competitor models.

The annotator will also support reinforcement learning workflows by building coding environments, validating code, and surfacing performance insights for model improvement.

Qualifications

• Strong programming in Python; experience with front-end languages is a plus.
• Experience in code review and QA processes.
• Ability to perform repetitive tasks with precision and attention to detail.

Responsabilités

• Create high-quality coding prompts and reference solutions aligned with benchmarks (SWE-Bench, Terminal-Bench).
• Develop prompts for code refactoring, generation, and problem solving.
• Evaluate model outputs to identify errors, limitations, and failure patterns in reasoning and execution.
• Design and maintain coding environments for evaluation and RL pipelines.
• Execute coding-specific validation checks using established criteria and tools.
• Document findings and insights to support model improvement and training strategies.
• Perform detailed code reviews and annotations for accuracy and compliance.
• Collaborate with cross-functional teams to improve code quality and documentation.

Connaissances

Python

JavaScript

TypeScript

Formation

Bachelor's degree in Computer Science / Software Engineering / Computer Engineering

Outils

Java

Description du poste

DataForce by TransPerfect in Paris, France is seeking Software Engineers to join as Coding Annotators to support AI model evaluation. You will create high-quality coding prompts and reference solutions, benchmark model performance, and identify failure cases across internal and competitor models.

The annotator will also support reinforcement learning workflows by building coding environments, validating code, and surfacing performance insights for model improvement.