Senior Software Engineer: Evaluation Pipelines (Remote)

Braintrust; United States; Remote; USD 114,000 - 162,000; Part time

Job summary

Braintrust is seeking a contract role to help an AI research team turn expert engineering judgment into reliable evaluation workflows. You will design evaluation rubrics, refine preference-data criteria, and build Python or TypeScript tooling to support data collection and evaluation.

This is a remote engagement for about three months, requiring at least 20 hours per week with four hours overlap with Pacific Time.

Qualifications

• At least five years of professional experience in Python pipelines or TypeScript in a large codebase.
• Hands-on experience collecting preference data or creating evaluation rubrics.
• Strong technical judgment and clear written and verbal communication.
• A current LinkedIn profile included on your resume

Responsibilities

• Design and refine evaluation rubrics and preference-data criteria.
• Review examples with a high bar for quality, explaining subtle issues and tradeoffs.
• Build or maintain Python or TypeScript tooling and pipelines that support data collection and evaluation.
• Turn findings into concise, actionable write-ups for researchers and engineers.

Skills

Python pipelines

TypeScript

Evaluation rubrics

Preference data

Communication

Job description

Braintrust is seeking a contract role to help an AI research team turn expert engineering judgment into reliable evaluation workflows. You will design evaluation rubrics, refine preference-data criteria, and build Python or TypeScript tooling to support data collection and evaluation.

This is a remote engagement for about three months, requiring at least 20 hours per week with four hours overlap with Pacific Time.