Senior LLM Evaluation Engineer & Coding Architect (Remote)

Braintrust; Paris; Hybride; EUR 110 000 - 150 000; Plein temps

Résumé du poste

Braintrust is seeking experienced software engineers for a contracting engagement to build and evaluate cutting-edge LLMs. Based in Paris or London, with Europe remote possible for strong candidates, the role sits at the intersection of real-world software engineering, model evaluation, and applied AI.

You'll design coding prompts, evaluate model outputs, document failures, and run head-to-head tests between private and external models.

Qualifications

• 10+ years of professional software development experience.
• Fluent in English (written and spoken).
• Strong Python skills (required).
• Knowledge of at least one additional programming language (bonus).
• 1+ year of coding annotation and/or LLM evaluation experience (part-time ok).
• Prior code reviewer experience is a plus.

Responsabilités

• Create high-quality coding prompts and reference answers (benchmark-style).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and RL workflows.
• Follow detailed annotation and evaluation guidelines with high consistency.

Connaissances

Python

Code evaluation

Team lead

English

Outils

Mistral-based LLMs

Description du poste

Braintrust is seeking experienced software engineers for a contracting engagement to build and evaluate cutting-edge LLMs. Based in Paris or London, with Europe remote possible for strong candidates, the role sits at the intersection of real-world software engineering, model evaluation, and applied AI.

You'll design coding prompts, evaluate model outputs, document failures, and run head-to-head tests between private and external models.