Member of Technical Staff, Agents

Judgment Labs

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Judgment Labs in San Francisco is seeking a hands-on technologist to own end-to-end product experiences that shape how researchers and engineers run large-scale investigations. You’ll talk to customers, define what to build, implement it, and iterate until it’s robust and useful.

You’ll design and ship an SDK and UI workflows that let Claude Code, Codex, and OpenCode sessions summon Judgment as a subagent, enabling parallel investigations, robust verification, and clear insights across complex

Qualifications

  • Experience building and scaling production systems end-to-end.
  • Strong problem-solving in ambiguous environments.
  • Hands-on with LLMs or agents, or quick to learn.

Responsibilities

  • Shape Judgment Agent operations across thousands of traces.
  • Build verification environments for agent changes.
  • Design interfaces to show engineers what agents did.
  • Create swarm UX to track parallel investigations.
  • Develop the improvement loop to convert trajectories into datasets and checks.
  • Manage platform aspects: workspaces, permissions, billing, usage.

Skills

End-to-end systems
LLMs/Agents
Customer interaction
Shipping quickly

Tools

LLM tooling

Job description

About Judgment

Judgment Labs is building the infrastructure for continual learning in long-horizon AI agents.

We turn long agent trajectories into clean, structured data for evals, labeling, rubric generation, context engineering, and RL workflows. Instead of only showing teams what happened, Judgment helps decide what matters, what should be learned from, and how that learning should flow back into the agent.

We’ve raised $32M across two rounds led by Lightspeed.

The Role

You'll build the product experiences that make this loop legible, and you'll build the agents that run it. This is not a role where you implement specs handed down. You'll own problems end-to-end: talking to customers, defining what to build, building it, and iterating until it's great.

What You’ll Do
  • Judgment Agent: Shape how the Judgment Agent runs large-scale investigations: parallel investigators working across thousands of production traces, each covering a different dimension (failure modes, tool errors, regressions, drift), merging results into one answer.

  • Verification: Build the platform for verifying agent changes: hosted simulated environments for stateful agent evals, trajectory replay against changed agents, and monitors for unintended behavior changes.

  • Agent investigation interfaces: Design how engineers understand what their agents did and why. Long traces, tool calls, decisions, failures. What does debugging look like when the "program" is a reasoning loop? How do you make a thousand-step trajectory legible in minutes?

  • Swarm UX: A hundred parallel investigations is useless if engineers can't follow them. Design how humans watch a swarm work, redirect investigators that go down the wrong path, and consume findings without reading a hundred reports.

  • The improvement loop: Build the workflows that turn production trajectories into datasets, judges, and regression checks, so the path from "found a problem" to "verified a fix" feels like one motion.

  • The platform underneath: Workspaces, roles, permissions, billing, usage, and limits for teams running many agents across many environments.

  • Judgment everywhere agents are built: An SDK and terminal-first experience so Claude Code, Codex, and OpenCode sessions can summon Judgment as a subagent mid-development.

What We’re Looking For
  • Experience building and scaling end-to-end production systems, from data layer to UI.

  • Strong technical problem-solving skills, especially in fast-changing, ambiguous environments.

  • A builder and tinkerer's mindset with high agency - you find creative ways to overcome obstacles and ship.

  • Hands-on experience building with LLMs or agents, or the drive to get there fast.

  • Comfort working directly with customers to understand their needs and solve real-world problems.

  • Excellent communication skills - clear, direct, and persuasive across technical and non-technical audiences.

  • Reinforcement learning, post-training, agents, or machine learning fundamentals.

Why Judgment?
  • You’ll work on problems customers actually feel: Engineers talk directly to teams building production agents, see where their systems fail, and turn those failures into product and infrastructure.

  • Small team, high ownership: You will not own a narrow slice. You’ll shape core systems early, ship quickly, and work across product, data, backend, infra, and customer environments.

  • In person in San Francisco: We work together in person because the problems are hard, the product is moving fast, and the feedback loops matter.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff, Product Engineer/Full Stack
Member of Technical Staff, Product Engineer/Full Stack

Judgment Labs Inc. • San Francisco (CA)

On-site
USD 140,000 - 230,000
Member of Technical Staff, Deployment Engineer
Member of Technical Staff, Deployment Engineer

Judgment Labs Inc. • San Francisco (CA)

On-site
USD 120,000 - 180,000
Full benefits
Equinox membership
Private chef
Member of Technical Staff, Deployment Engineer
Member of Technical Staff, Deployment Engineer

Judgment, Inc. • San Francisco (CA)

On-site
USD 120,000 - 180,000
Full benefits
Equinox membership
Private chef
+1
Founding Business Development Representative
Founding Business Development Representative

Judgment, Inc. • San Francisco (CA)

On-site
USD 90,000 - 150,000
Staff Engineer, AI Agent Platform & Investigations
Staff Engineer, AI Agent Platform & Investigations

Judgment Labs • San Francisco (CA)

On-site
USD 180,000 - 240,000
Staff Software Engineer, Agent Eval Platform
Staff Software Engineer, Agent Eval Platform

ServiceNow • Santa Clara (CA)

On-site
USD 180,000 - 320,000
Software Engineer
Software Engineer

Turing • Alexandria (VA)

On-site
USD 120,000 - 180,000
Equity
Senior Software Engineer
Senior Software Engineer

vibehackers • United States

Hybrid
USD 140,000 - 210,000
Senior Engineer, AI and Agents Remote / Hybrid (flexible)
Senior Engineer, AI and Agents Remote / Hybrid (flexible)

Falcon Labs • United States

Hybrid
USD 150,000 - 210,000
Senior Engineering Manager
Senior Engineering Manager

Mixpeek • San Francisco (CA)

On-site
USD 170,000 - 250,000