Senior LLM Evaluation Engineer & Coding Specialist

Braintrust; Netherlands; On-site; EUR 90,000 - 150,000; Part time

Job summary

Braintrust is seeking seasoned software engineers for a contracting role focused on evaluation and annotation of state-of-the-art LLMs. You will design coding prompts, assess model outputs, and document failures while working at a senior level with a focused, highly capable team.

Ideal candidates have 10+ years in software development, expert Python skills, and hands-on experience with code evaluation or LLM benchmarking.

Qualifications

• 10+ years of professional software development experience.
• Strong Python skills (required).
• Knowledge of at least one additional programming language.
• 1+ year of coding annotation and/or LLM evaluation experience (part-time OK) for a major frontier AI lab or AI infrastructure company.
• Prior code reviewer experience is a plus.
• Proven ability to apply structured evaluation criteria and write clear technical feedback.
• Fluent in English (written and spoken).
• Team lead or mentoring experience is a strong plus.

Responsibilities

• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

Python

English fluency

Other programming language

Job description

Braintrust is seeking seasoned software engineers for a contracting role focused on evaluation and annotation of state-of-the-art LLMs. You will design coding prompts, assess model outputs, and document failures while working at a senior level with a focused, highly capable team.

Ideal candidates have 10+ years in software development, expert Python skills, and hands-on experience with code evaluation or LLM benchmarking.