Staff Product Manager, AI Coding Evals & Benchmarks

Hercules; Stati Uniti; Ibrido; USD 200.000 - 350.000; Tempo pieno

Vantaggi offerti da questo lavoro

Unlimited AI token budget

Take what you need vacation policy

Healthcare, vision, and dental

Daily lunches, dinners and snacks in t

We guarantee you will do the best work

Descrizione del lavoro

Hercules seeks a senior or staff product manager to own evals for our AI coding agent. You will drive offline and online evals, shaping how customers experience our coding assistant.

You will leverage 500k+ users, 1M+ code repos, and millions of chat threads to craft effective evaluation methodologies while ensuring privacy and security. This role requires deep technical fluency and product intuition.

Competenze

• Strong across listed areas including software fundamentals and evals.
• Experience creating and/or managing technical evals.
• Excellent product and design sense for user goals.
• Experience with AI workflows and frontier technologies.

Mansioni

• Own offline and online evals for Hercules' AI coding agent.
• Make the AI coding agent cheaper, faster, and more accurate.
• Leverage 500k+ users, 1M+ repos, and millions of chat threads to guide eval creation.
• Anonymise data to protect customer privacy.

Conoscenze

Software fundamentals

Eval methodologies

Product sense

Design sense

AI context & engineering

Strumenti

TypeScript

React

Agent framework / AI SDK

Eval platform (LangSmith)

Descrizione del lavoro

Hercules seeks a senior or staff product manager to own evals for our AI coding agent. You will drive offline and online evals, shaping how customers experience our coding assistant.

You will leverage 500k+ users, 1M+ code repos, and millions of chat threads to craft effective evaluation methodologies while ensuring privacy and security. This role requires deep technical fluency and product intuition.