Senior Coding & AI Model Evaluation Engineer

Braintrust; United States; Remote; USD 103,000 - 124,000; Part time

Job summary

Braintrust is seeking experienced software engineers for a contracting engagement focused on evaluating and annotating AI models. You will design coding tasks, assess model outputs for code-related tasks, and document failures with rigorous feedback.

This is a senior, hands-on role requiring strong engineering judgment and language fluency. The role centers on real-world software engineering, model evaluation, and applied AI, with a focus on high-quality benchmarking and RL workflows.

Qualifications

• 5+ years of professional software development experience.
• Strong Python skills (required).
• Knowledge of at least one additional programming language.
• Experience with professional code review, coding annotation, LLM/code evaluation, or benchmark design is a plus, but not required.
• Hands-on experience with vulnerability research, exploit reproduction or verification, or implementing security patches.
• Proven ability to apply structured evaluation criteria and write clear technical feedback.
• Fluent in English (written and spoken).
• Team lead or mentoring experience is a strong plus.

Responsibilities

• Evaluate coding tasks involving software vulnerabilities, exploit verification, and security patches.
• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

5+ years software development

Python

Other language

Code review / evaluation

Vulnerability research

English fluency

Team leadership

Job description

Braintrust is seeking experienced software engineers for a contracting engagement focused on evaluating and annotating AI models. You will design coding tasks, assess model outputs for code-related tasks, and document failures with rigorous feedback.

This is a senior, hands-on role requiring strong engineering judgment and language fluency. The role centers on real-world software engineering, model evaluation, and applied AI, with a focus on high-quality benchmarking and RL workflows.