Product Engineer - AI Judgment Platform

Judgment Labs; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Judgment Labs looks for a Product Engineer who will own end-to-end production experiences for AI agents. You’ll shape how large-scale investigations run, and build verification environments with trajectory replay to ensure changes are correct and robust.

You’ll also design interfaces for engineers to understand agent behavior, while enabling a swarm-style view of parallel investigations and repeatable workflows.

Qualifications

• Experience building and scaling end-to-end production systems.
• Strong problem-solving skills in fast-changing, ambiguous environments.
• Hands-on experience with LLMs or agents.
• Comfort talking to customers to understand needs and translate into features.
• Excellent communication across technical and non-technical audiences.

Responsibilities

• Shape how Judgment Agent runs large-scale investigations across thousands of production traces and dimensions.
• Build the platform for verifying agent changes with simulated environments and trajectory replay.
• Design interfaces to show engineers what their agents did and why.
• Create Swarm UX so multiple investigations can be followed and steered efficiently.
• Develop workflows turning trajectories into datasets, judges, and regression checks.
• Oversee the platform aspects: workspaces, roles, permissions, billing, usage.
• Provide an SDK/terminal-first experience for agent sessions to summon Judgment during development.

Skills

End-to-end production systems

LLMs / agents

Customer-facing communication

Problem solving

Rapid iteration

Job description

Judgment Labs looks for a Product Engineer who will own end-to-end production experiences for AI agents. You’ll shape how large-scale investigations run, and build verification environments with trajectory replay to ensure changes are correct and robust.

You’ll also design interfaces for engineers to understand agent behavior, while enabling a swarm-style view of parallel investigations and repeatable workflows.