Staff Technical Product Manager, Evals

Digitailor; San Francisco (CA); On-site; USD 200,000 - 350,000; Full time

Benefits offered by this job

Unlimited AI token budget

Healthcare, vision, and dental

Daily meals in office

Job summary

Digitailor? Hercules is seeking a Staff Technical Product Manager to own evals for the AI coding agent in San Francisco. This role focuses on making the evals cheaper, faster, and more accurate by leveraging a massive user base and data repositories.

You will work closely with engineering on evaluation design and privacy-conscious data handling. The ideal candidate combines deep software and AI context experience with product sense to craft intuitive UX and scalable eval platforms, using

Qualifications

• Senior or staff-level product manager with ownership of evals for an AI coding agent.
• Strong technical background in software fundamentals, system design, data modeling, and API design.
• Experience creating and managing technical evals at scale.

Responsibilities

• Own Hercules Bench and Hercules online evals to improve cost, speed, and accuracy.
• Leverage large user and code repo datasets to guide evals.
• Collaborate with engineering on complex eval implementations.
• anonymize data to protect customer privacy.

Skills

Software fundamentals

System design

Data modeling

Architecture

Coding

Product sense

Design sense

AI context

TypeScript

React

API design

Eval platforms

AI SDK

Postgres

Redis

Terraform

Docker

Education

Bachelor's degree in a technical field

Tools

LangSmith

OpenAI / Anthropic / Google APIs

Job description

# Staff Technical Product Manager, EvalsHercules · San Francisco, California, United States · posted on AshbyThis posting does not mention sponsorship either way. Hercules has 9 certified H-1B filings on record with the US Department of Labor. That is the employer’s history, not a statement about this job.Sign up to applyDigitailor tailors your resume to this posting and fills the employer’s form with you watching. That needs an account.What are we looking for? Hercules is looking for a senior or staff level product manager to own evals for our AI coding agent What will you do? Your #1 and only goal is to make Hercules' AI coding agent cheaper, faster, and more accurate (our pareto frontier) You will own Hercules Bench, our offline eval set You will own Hercules online evals You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to guide your eval creation You will anonymize data to protect our customer's privacy What skills do we expect you to have? We expect candidates to be incredibly strong across all of the following areas: Software fundamentals : You are building an AI coding agent. We expect you to be technical and able to code. We expect you to have a decent understanding of system design, architecture, data modeling, and API design. In saying this, you will work with an engineering counterpart to work on complex implementation Evals . You have explicitly created and/or managed technical evals before. You should be able to work across a variety of different eval methodologies and deeply understand optimal reward modeling, Product sense : strong intuition for the goals customers are trying to achieve and novel solutions to help them achieve them Design sense: Track record of building simple, delightful UX/UI that makes it easy for customers to achieve their goals AI context and harness engineering : strong experience with the most up to date context and harness engineering best practices like summarization/compression, prompting, prompt caching, subagents, memory, context management, tools, async tools, deferred tools, skills, etc Yes, we know this industry is evolving rapidly and the \"best practices\" change every week, but yes, we are still actively looking for someone who has wrestled with these challenges at scale and has strong opinions about what works and what doesn't Our core stack : TypeScript, React, (some sort of agent framework like AI SDK), (some sort of eval platform like LangSmith) AI : able to truly 10x your workflow by being on frontier of AI coding What’s Hercules’ stack? Technologies: Typescript, React, AI SDK, Postgres, Redis, Clickhouse, Tailwind, IaC (CDK, Terraform), Docker, Otel, Hono, Drizzle ORM, Temporal Clouds: Cloudflare, AWS AI Models: OpenAI, Anthropic, Google What are Hercules’ operating principles? Hercules’ operating principles guide how we make decisions every day. We take them very seriously. Most of the principles can be learned on the job, however, we do specifically filter for the following principles in our hiring process: We ship . We ship fast, iterate, and correct mistakes quickly. Our bias for action speeds up our learning and delights our customers. We strive for excellence. Doing things well is in our DNA. We have good taste. We get the details right. Our outputs are exceptionally simple and self-serve. We’re proud of our work. We are hungry . We are willing to work hard and break through walls to achieve the outcomes we want. We are willing to operate at any level to get things done. Is Hercules in-office or remote? Hercules founding team works in-office in San Francisco (Kearny and Bush). We have a very strong preference for in office. We are open to remote for exceptional candidates who are willing to 1) work in US time zones and 2) travel to the office regularly. What are Hercules’ benefits? Unlimited AI token budget Take what you need vacation policy Healthcare, vision, and dental Daily lunches, dinners and snacks in the office We guarantee you will do the best work of your career here What’s the compensation? $200k-$350k + equity (variable based on seniority, equity, and location)Read from Ashby · posted 2026-10-21. See every open role.