Principal Coding Annotator / LLM Evaluation Engineer

Braintrust; United States; Remote; USD 103,000 - 124,000; Part time

Job summary

Braintrust is seeking experienced software engineers for a contracting engagement focused on evaluating and annotating AI models. You will design coding tasks, assess model outputs for code-related tasks, and document failures with rigorous feedback.

This is a senior, hands-on role requiring strong engineering judgment and language fluency. The role centers on real-world software engineering, model evaluation, and applied AI, with a focus on high-quality benchmarking and RL workflows.

Qualifications

• 5+ years of professional software development experience.
• Strong Python skills (required).
• Knowledge of at least one additional programming language.
• Experience with professional code review, coding annotation, LLM/code evaluation, or benchmark design is a plus, but not required.
• Hands-on experience with vulnerability research, exploit reproduction or verification, or implementing security patches.
• Proven ability to apply structured evaluation criteria and write clear technical feedback.
• Fluent in English (written and spoken).
• Team lead or mentoring experience is a strong plus.

Responsibilities

• Evaluate coding tasks involving software vulnerabilities, exploit verification, and security patches.
• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

5+ years software development

Python

Other language

Code review / evaluation

Vulnerability research

English fluency

Team leadership

Job description

• Rate: $75 – $90/hr
• Hours: 20 hours / week
• Experience: 10+ years
• Location: Belgium | Denmark + 10 more

This is a contracting engagement - initially 6 months - with potential for long term engagement.

Location: Paris or London-based preferred; alternatively Europe remote for strong candidates

We are building and evaluating state-of-the-art large language models (LLMs) and are looking for experienced software engineers to join our evaluation and annotation team. This role sits at the intersection of real-world software engineering, model evaluation, and applied AI , and is critical to improving model reliability, reasoning, and code quality.

You will design challenging coding tasks, evaluate model outputs against rigorous benchmarks, identify failure modes, and contribute to reinforcement learning and model improvement workflows.

This is not a junior annotation role. We are looking for practitioners with deep hands-on coding experience who can think like both an engineer and an evaluator.

What You’ll Do

• Evaluate coding tasks involving software vulnerabilities, exploit verification, and security patches.
• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.

What We’re Looking For

• 5+ years of professional software development experience.
• Strong Python skills (required).
• Knowledge of at least one additional programming language (bonus).
• Experience with professional code review, coding annotation, LLM/code evaluation, or benchmark design is a plus, but not required.
• Hands-on experience with vulnerability research, exploit reproduction or verification, or implementing, backporting, or validating security patches.
• Proven ability to apply structured evaluation criteria and write clear technical feedback.
• Fluent in English (written and spoken).
• Team lead or mentoring experience is a strong plus.

Why This Role

• Work hands-on with cutting-edge LLMs.
• Apply real-world engineering judgment to model evaluation and improvement.
• High-impact, technical work with a focused, senior team.

Company: Leading AI Lab