Evals Platform Engineer — Build AI Evaluation & Benchmarks

Mercor; San Francisco (CA); Presencial; USD 140 000 - 190 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Bi-annual performance bonus

Equity grant

Relocation bonus

Housing stipend

Meals stipend

Health insurance

Wellness reimbursement

Resumo da oferta

Mercor in San Francisco is seeking a Software Engineer for the Evals product team who will build the scorecards, failure reports, and dataset views that help customers turn eval runs into decisions. You will ship APIs, data models, and UI, own APEX launches, and develop explanations for model failures while turning customer feedback into the roadmap.

The role grows with the platform, working across evals, post-training, and new product areas, and involves direct collaboration with labs and

Qualificações

• Bachelor's degree in CS or related field.
• Strong full-stack skills with Python/Go/TypeScript and React.
• Track record of owning features end to end, from idea to production.
• Strong product sense starting from user needs.
• Experience building data-heavy products: data models, APIs and views.
• Excellent written and verbal communication with engineers, researchers and customers.
• Comfort with ambiguity and iterative shipping.
• Experience with LLM evals, benchmarks or AI developer tools is a plus.
• Startup or B2B developer product experience is a plus.

Responsabilidades

• Build the product customers use to evaluate models: APIs, data models, and UI.
• Own APEX launches: run new frontier models and publish results to the leaderboard.
• Explain why models fail: develop analyses linked to concrete steps that went wrong.
• Make benchmarks easy to find and manage: dataset views, registries, versioning.
• Take customers from first login to self-serve: onboarding, roles and usage views.
• Work directly with customers: participate in pilots and feed roadmap.

Conhecimentos

Full-stack development

Python

TypeScript

React

Formação académica

Bachelor's Degree in CS

Ferramentas

React

Descrição da oferta de emprego

Mercor in San Francisco is seeking a Software Engineer for the Evals product team who will build the scorecards, failure reports, and dataset views that help customers turn eval runs into decisions. You will ship APIs, data models, and UI, own APEX launches, and develop explanations for model failures while turning customer feedback into the roadmap.

The role grows with the platform, working across evals, post-training, and new product areas, and involves direct collaboration with labs and