AI Platform PM: Orchestration, Evaluation & Safety

Disco, Inc.; New York, Northern (NY, KY); Hybrid; USD 190,000 - 220,000; Full time

Benefits offered by this job

Flexible PTO

12 paid holidays

DisConnect Days

$250/month lifestyle stipend

$100/month Disco credit

$750/month childcare stipend

$500 home office setup reimbursement

$20 daily Grubhub credit

401(k)

Job summary

Disco, Inc. is hiring an AI/ML Product Manager to own the end-to-end AI platform, ensuring model and harness agnosticism while optimizing cost and performance.

You will own the eval loops, safety guardrails, and the agentic pipeline that enables a team of 25 to move with the velocity of a larger group. You’ll work across product, eng, and data teams to redesign orchestration layers, compare models and harnesses, and dive into data and code to identify token usage issues and workflow bottlenecks.

Qualifications

• Proven track record shipping AI/ML products to production.
• Ability to read code, query data, and reason about model tradeoffs.
• Strong product judgment with a roadmap mindset.
• Hybrid NYC office-based role with willingness to collaborate in person.

Responsibilities

• Own the AI platform routing layer, ensuring model- and harness-agnostic operation.
• Manage eval loops, guardrails, and cost oversight for model usage.
• Architect and improve the agentic pipeline, from deployment to monitoring.
• Collaborate with engineers, data scientists, and product teams to ship safe AI features.
• Define success metrics and drive continuous improvement through data-driven decisions.

Skills

AI product management

Cross-functional leadership

System design & orchestration

Model evaluation & guardrails

Budgeting & cost awareness

Technical literacy & code reading

Tools

AWS Bedrock

AI tooling & automation

Job description

Disco, Inc. is hiring an AI/ML Product Manager to own the end-to-end AI platform, ensuring model and harness agnosticism while optimizing cost and performance.

You will own the eval loops, safety guardrails, and the agentic pipeline that enables a team of 25 to move with the velocity of a larger group. You’ll work across product, eng, and data teams to redesign orchestration layers, compare models and harnesses, and dive into data and code to identify token usage issues and workflow bottlenecks.