AI Code Evaluation Security Engineer — Remote Contract

Braintrust; Portugal; Remote; EUR 70,000 - 120,000; Full time

Job summary

Braintrust is seeking a seasoned software engineer to help a top AI lab evaluate and improve large language models through security-focused coding tasks. You will evaluate vulnerabilities, reproduce exploits, verify patches, and create prompts and reference answers; assess model outputs for code generation, debugging and refactoring.

Remote contracting with international eligibility and flexible engagement terms.

Qualifications

• At least five years of professional software development experience and strong Python skills.
• Hands-on experience with vulnerability research, exploit reproduction or verification, or implementing security patches.
• Ability to apply structured evaluation criteria and provide clear technical feedback.

Responsibilities

• Evaluate coding tasks involving vulnerabilities, exploit verification and security patches.
• Create high-quality coding prompts and reference answers for benchmark problems.
• Evaluate model outputs for code generation, refactoring, debugging and implementation.
• Identify and document model failures, edge cases and reasoning gaps.
• Compare private language models with leading external models.
• Build or configure coding environments for evaluation and reinforcement learning.
• Follow detailed annotation and evaluation guidelines consistently.

Skills

Python

Vulnerability research

Exploit verification

Security patches

English fluency

Job description

Braintrust is seeking a seasoned software engineer to help a top AI lab evaluate and improve large language models through security-focused coding tasks. You will evaluate vulnerabilities, reproduce exploits, verify patches, and create prompts and reference answers; assess model outputs for code generation, debugging and refactoring.

Remote contracting with international eligibility and flexible engagement terms.