Staff Technical PM — AI Coding Evals & Platform

Hercules; San Francisco (CA); On-site; USD 200,000 - 350,000; Full time

Benefits offered by this job

Unlimited AI token budget

Take what you need vacation policy

Healthcare, vision, and dental

Daily lunches, dinners and snacks in,?

Job summary

Hercules is seeking a senior or staff product manager to own evals for our AI coding agent in San Francisco. You will direct Hercules Bench offline evals and Hercules online evals, shaping how we measure performance using our huge user and code repository base.

You will craft evaluation methodologies, balance speed, cost, and accuracy, and ensure privacy through data anonymization while working with core stack technologies like TypeScript and React.

Qualifications

• Strong technical coding ability and system design understanding.
• Experience creating and/or managing technical evals.
• Strong product intuition for customer goals and solutions.
• Track record of simple, delightful UX/UI design.

Responsibilities

• Own evals for Hercules' AI coding agent.
• Own Hercules Bench (offline eval set).
• Own Hercules online evals.
• Leverage 500k+ users, 1M+ code repos, and millions of chat threads to guide eval creation.
• Anonymize data to protect customer privacy.

Skills

Software fundamentals

Eval design

Product sense

Design sense

AI context

Core stack

AI impact

Tools

TypeScript

React

AI SDK

LangSmith

Job description

Hercules is seeking a senior or staff product manager to own evals for our AI coding agent in San Francisco. You will direct Hercules Bench offline evals and Hercules online evals, shaping how we measure performance using our huge user and code repository base.

You will craft evaluation methodologies, balance speed, cost, and accuracy, and ensure privacy through data anonymization while working with core stack technologies like TypeScript and React.