Software Engineer, Evals Platform — AI Benchmarking

mercor.com; San Francisco (CA); Presencial; USD 150 000 - 200 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Bi-annual performance bonus

Equity grant vesting over 4 years

Relocation bonus up to $15k

Housing bonus up to $10K (within 0.5mi

Meal stipend $1.5K monthly

Equinox membership

Laundry reimbursement $200 monthly

Wellness reimbursement $200 monthly

Health, Dental, Vision insurance

Resumo da oferta

Mercor in San Francisco is hiring a Software Engineer for the Evals product team. You will build scorecards, failure reports and dataset views to help customers turn eval runs into actionable decisions. You will address scale, grading accuracy and reproducibility while keeping the product simple to use.

The role grows with the platform, expanding from evals into post-training and new product areas. You will transform failures into data that improves models and drive impactful outcomes for

Qualificações

• BA/BS in Computer Science or related field.
• Strong full-stack skills in a modern backend language (Python, Go or TypeScript).
• Frontend experience with React or similar.
• Track record of owning product features end-to-end, from idea to production.
• Strong product sense and user-focused thinking.
• Experience building data-heavy products: data models, APIs and views over large results.
• Excellent written and verbal communication with engineers, researchers and customers.
• Comfort with ambiguity and a habit of shipping iteratively.
• Experience with LLM evals, benchmarks or AI developer tools is a plus.
• Startup experience or experience building B2B developer products is a plus.

Responsabilidades

• Build the product customers use to evaluate models: Ship APIs, data models and UI for running evals, comparing models and sharing scores.
• Own APEX launches: Run new frontier models on APEX and publish correct, defensible results to the public leaderboard.
• Explain why models fail: Build failure analysis tied to exact steps that went wrong.
• Make benchmarks easy to find and manage: Build dataset views, registries and versioning.
• Take customers from first login to self-serve: Build onboarding, roles, key management and usage views.
• Work directly with customers: Join pilots with labs and enterprises and translate needs into the roadmap.

Conhecimentos

Full-stack skills

Backend languages (Python/Go/TypeScrip

Frontend: React

Own product features end-to-end

Product sense

Data-heavy products

Communication

Shipping iteratively

LLM evals / AI tools (plus)

Startup/B2B product experience (plus)

Formação académica

Bachelor's Degree in Computer Science or related field

Ferramentas

Python

Go

TypeScript

React

Descrição da oferta de emprego

Mercor in San Francisco is hiring a Software Engineer for the Evals product team. You will build scorecards, failure reports and dataset views to help customers turn eval runs into actionable decisions. You will address scale, grading accuracy and reproducibility while keeping the product simple to use.

The role grows with the platform, expanding from evals into post-training and new product areas. You will transform failures into data that improves models and drive impactful outcomes for