Senior AI Evaluation Product Manager
Speedrun Talent Network
A complete application in a minute
Tailored resume and cover letter, ready to send.
Benefits offered by this job
Job summary
Hercules is seeking a senior or staff level product manager to own evals for our AI coding agent. You will own Hercules Bench offline evals and Hercules online evals, guiding evaluation design and data-driven improvements.
You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to craft effective evals while anonymizing data to protect privacy. We ship, strive for excellence, and stay hungry for impact.
Qualifications
- Software fundamentals: technical, coding, architecture, API design.
- Eval design: created/managed technical evals, reward modeling.
- Product sense: intuition for customer goals and solutions.
- Design sense: simple, delightful UX/UI.
- AI context & harness engineering: summarization, prompting, memory, tools.
- Core stack: TypeScript, React, AI SDK, LangSmith-like eval platform.
- AI: able to 10x workflow at frontier of AI coding.
Responsibilities
- Own evals for Hercules' AI coding agent.
- Own Hercules Bench offline eval set.
- Own Hercules online evals.
- Leverage 500k+ users, 1M+ repos, and millions of chats to guide eval creation.
- Anonymize data to protect customer privacy.
Skills
Tools
Job description
Hercules is seeking a senior or staff level product manager to own evals for our AI coding agent. You will own Hercules Bench offline evals and Hercules online evals, guiding evaluation design and data-driven improvements.
You will leverage our 500k+ users, 1M+ code repos, and millions of chat threads to craft effective evals while anonymizing data to protect privacy. We ship, strive for excellence, and stay hungry for impact.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
