Staff AI Evals Product Manager

Digitailor; San Francisco (CA); On-site; USD 200,000 - 350,000; Full time

Benefits offered by this job

Unlimited AI token budget

Healthcare, vision, and dental

Daily meals in office

Job summary

Digitailor? Hercules is seeking a Staff Technical Product Manager to own evals for the AI coding agent in San Francisco. This role focuses on making the evals cheaper, faster, and more accurate by leveraging a massive user base and data repositories.

You will work closely with engineering on evaluation design and privacy-conscious data handling. The ideal candidate combines deep software and AI context experience with product sense to craft intuitive UX and scalable eval platforms, using

Qualifications

• Senior or staff-level product manager with ownership of evals for an AI coding agent.
• Strong technical background in software fundamentals, system design, data modeling, and API design.
• Experience creating and managing technical evals at scale.

Responsibilities

• Own Hercules Bench and Hercules online evals to improve cost, speed, and accuracy.
• Leverage large user and code repo datasets to guide evals.
• Collaborate with engineering on complex eval implementations.
• anonymize data to protect customer privacy.

Skills

Software fundamentals

System design

Data modeling

Architecture

Coding

Product sense

Design sense

AI context

TypeScript

React

API design

Eval platforms

AI SDK

Postgres

Redis

Terraform

Docker

Education

Bachelor's degree in a technical field

Tools

LangSmith

OpenAI / Anthropic / Google APIs

Job description

Digitailor? Hercules is seeking a Staff Technical Product Manager to own evals for the AI coding agent in San Francisco. This role focuses on making the evals cheaper, faster, and more accurate by leveraging a massive user base and data repositories.

You will work closely with engineering on evaluation design and privacy-conscious data handling. The ideal candidate combines deep software and AI context experience with product sense to craft intuitive UX and scalable eval platforms, using