Senior LLM Evaluation & Coding Engineer (Remote EU)

Braintrust; United Kingdom; On-site; GBP 70,706 - 141,412; Full time

Job summary

Braintrust is hiring experienced software engineers to join our evaluation and annotation team for a contracting engagement. The role focuses on real-world software engineering, model evaluation, and applied AI, aiming to improve model reliability, reasoning, and code quality.

You will design challenging coding tasks, evaluate model outputs against rigorous benchmarks, identify failure modes, and contribute to reinforcement learning and model improvement workflows.

Qualifications

• 10+ years of professional software development experience.
• Strong Python skills are required.
• At least 1 year of coding annotation and/or LLM evaluation experience.
• Ability to lead or mentor others is a plus.
• Fluent in English (written and spoken).

Responsibilities

• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

Python

Code review

LLM evaluation

English fluency

Team leadership

Tools

Git

Python tooling

Job description

Braintrust is hiring experienced software engineers to join our evaluation and annotation team for a contracting engagement. The role focuses on real-world software engineering, model evaluation, and applied AI, aiming to improve model reliability, reasoning, and code quality.

You will design challenging coding tasks, evaluate model outputs against rigorous benchmarks, identify failure modes, and contribute to reinforcement learning and model improvement workflows.