Senior Coding Annotator & LLM Evaluation Engineer - Remote

Braintrust; Town of Belgium (WI); On-site; USD 104,000 - 173,000; Part time

Job summary

Braintrust is seeking an experienced software engineer to join our evaluation and annotation team, focusing on real-world coding tasks and model evaluation. This role requires hands-on coding expertise, the ability to think like an engineer and evaluator, and strong English communication.

The engagement is contracting for 6 months with potential for extension. Location options include Paris/London or Europe remote for top candidates, offering a high-impact, senior team environment focused on

Qualifications

• 10+ years of professional software development experience.
• Strong Python skills (required).
• Knowledge of at least one additional programming language (bonus).
• 1+ year of coding annotation and/or LLM evaluation experience (part-time OK).
• Prior code reviewer experience is a plus.
• Fluent in English (written and spoken).
• Team lead or mentoring experience is a strong plus.

Responsibilities

• Create high-quality coding prompts and reference answers (benchmark-style).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and RL.
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

Python

LLM evaluation

Code review

Benchmark design

English fluency

Tools

Mistral-based LLMs

Job description

Braintrust is seeking an experienced software engineer to join our evaluation and annotation team, focusing on real-world coding tasks and model evaluation. This role requires hands-on coding expertise, the ability to think like an engineer and evaluator, and strong English communication.

The engagement is contracting for 6 months with potential for extension. Location options include Paris/London or Europe remote for top candidates, offering a high-impact, senior team environment focused on