Senior AI Code Evaluator & Task Designer (Remote)

Braintrust; United States; Remote; USD 96,000 - 117,000; Full time

Job summary

Leading AI Lab is seeking experienced software engineers to join an evaluation and annotation team focused on real-world software engineering, model evaluation, and AI-driven improvements.

Ideal candidates have deep hands-on coding experience, strong Python skills, and a track record in structured evaluation and high-quality feedback. This is a senior, contract-friendly role with collaboration across private LLMs and external models.

Qualifications

• 5+ years of professional software development experience.
• Strong Python skills.
• Knowledge of at least one additional programming language.
• 1+ year of coding annotation and/or LLM evaluation experience.
• Prior code reviewer experience is a plus.
• Fluent in English (written and spoken).
• Team lead or mentoring experience is a strong plus.

Responsibilities

• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.

Skills

Python

Code review

Team leadership

Job description

Leading AI Lab is seeking experienced software engineers to join an evaluation and annotation team focused on real-world software engineering, model evaluation, and AI-driven improvements.

Ideal candidates have deep hands-on coding experience, strong Python skills, and a track record in structured evaluation and high-quality feedback. This is a senior, contract-friendly role with collaboration across private LLMs and external models.