Senior LLM Evaluation Engineer & Coding Annotator

Braintrust; Germany (OH); On-site; USD 111,032 - 190,341; Full time

Job summary

Braintrust seeks an experienced software engineer to join our evaluation and annotation team. You will work at the intersection of real-world software engineering, model evaluation, and applied AI, contributing to reliability and code quality.

This contracting engagement is initially six months with potential for long-term engagement. Location preferences are Paris or London, with Europe-wide remote options for highly qualified candidates.

Qualifications

• 10+ years of professional software development experience.
• Strong Python skills required.
• 1+ year of coding annotation and/or LLM evaluation experience (part-time OK).
• Fluent in English (written and spoken).

Responsibilities

• Create high-quality coding prompts and reference answers (benchmark-style).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and RL.
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

Python

Software development

Code evaluation / LLM evaluation

English fluency

Team leadership

Mentoring

Code review

LLM evaluation tools

Education

Bachelor's degree in CS or related field

Tools

Mistral-based LLMs

Job description

Braintrust seeks an experienced software engineer to join our evaluation and annotation team. You will work at the intersection of real-world software engineering, model evaluation, and applied AI, contributing to reliability and code quality.

This contracting engagement is initially six months with potential for long-term engagement. Location preferences are Paris or London, with Europe-wide remote options for highly qualified candidates.