Senior Application Security Engineer - AI Code Evaluation

Braintrust; United States; Remote; USD 103,000 - 124,000; Full time

Job summary

Leading AI Lab is seeking a contractor to evaluate security-focused coding tasks and improve LLM performance. Remote candidates in the United States and other eligible locations may participate for a 20 hours per week commitment at $75–$90 per hour.

You will bring 5+ years of software development and strong Python skills, along with hands-on vulnerability research, exploit verification, and security patch experience. Fluency in English is required.

Qualifications

• Minimum 5 years of professional software development experience.
• Strong Python skills required.
• Hands-on vulnerability research or exploit verification experience.
• Ability to document findings with clear technical feedback.
• Fluent in written and spoken English.

Responsibilities

• Evaluate coding tasks involving vulnerabilities, exploit verification and security patches.
• Create high-quality coding prompts and reference answers for benchmarks.
• Evaluate model outputs for code generation, refactoring, debugging and implementation.
• Identify and document model failures, edge cases and reasoning gaps.
• Compare private language models with leading external models.
• Build or configure coding environments for evaluation and reinforcement learning.
• Follow detailed annotation and evaluation guidelines consistently.

Skills

5+ years software development

Python

Vulnerability research

Exploit verification

Security patches

English fluency

Tools

Python

Job description

• Rate: $75 – $90/hr
• Hours: 20 hours / week
• Experience: 5 - 10 years
• Location: United States | Canada + 21 more

Help a top AI lab evaluate and improve large language models through security-focused coding tasks. Bring your software-engineering judgment and hands‑on security experience to work involving vulnerabilities, exploit verification and security patches.

This is a contracting engagement, with potential for a longer‑term engagement. Remote candidates in the selected countries are elegible.

What you will do

• Evaluate coding tasks involving software vulnerabilities, exploit verification and security patches.
• Create high‑quality coding prompts and reference answers for benchmark‑style problems.
• Evaluate model outputs for code generation, refactoring, debugging and implementation.
• Identify and document model failures, edge cases and reasoning gaps.
• Compare private language models with leading external models.
• Build or configure coding environments for evaluation and reinforcement learning.
• Follow detailed annotation and evaluation guidelines consistently.

What you bring

• At least five years of professional software‑development experience and strong Python skills.
• Hands‑on experience with vulnerability research, exploit reproduction or verification, or implementing, backporting or validating security patches.
• The ability to apply structured evaluation criteria and write clear technical feedback.
• Fluency in written and spoken English.

Helpful, not required

• Professional code review, coding annotation, LLM/code evaluation or benchmark design.
• Knowledge of another programming language.
• Team leadership or mentoring experience.

Company: Leading AI Lab