Senior LLM Evaluation Engineer - Code & AI Systems

Braintrust; Greater London; On-site; GBP 90,000 - 150,000; Full time

Job summary

Braintrust is seeking an experienced software engineer to join our evaluation and annotation team. The role focuses on real-world software engineering, model evaluation, and applied AI to improve model reliability, reasoning, and code quality.

You will design challenging coding tasks, evaluate model outputs against benchmarks, and contribute to reinforcement learning and model improvement workflows. This is a contracting engagement with potential for long-term collaboration.

Qualifications

• 10+ years of professional software development experience.
• Strong Python skills (required).
• Knowledge of at least one additional programming language (bonus).
• 1+ year of coding annotation and/or LLM evaluation experience (part-time OK).
• Prior code reviewer experience is a plus.
• Fluent in English (written and spoken).
• Team lead or mentoring experience is a strong plus.

Responsibilities

• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

Python

Code review

Mentoring

LLM evaluation

Job description

Braintrust is seeking an experienced software engineer to join our evaluation and annotation team. The role focuses on real-world software engineering, model evaluation, and applied AI to improve model reliability, reasoning, and code quality.

You will design challenging coding tasks, evaluate model outputs against benchmarks, and contribute to reinforcement learning and model improvement workflows. This is a contracting engagement with potential for long-term collaboration.