Staff AI Systems Engineer: Evaluation Pipelines & Agents

NextToken; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Stock options

Comprehensive health plan

Gym stipend

Job summary

NextToken is seeking a Member of Technical Staff in San Francisco to build and scale production AI systems. You will own end-to-end evaluation pipelines, harnesses for agents, post-training workflows, and interfaces that turn experimental results into deployable improvements.

The role emphasizes model-agnostic reasoning, metric-driven development, and hands-on work with leading AI tooling to optimize latency, cost, and quality at scale.

Qualifications

• 3+ years building production AI systems.
• Experience building eval harnesses from scratch.
• Ability to reason about prompts, context and models.
• Proactive problem solver with high initiative.

Responsibilities

• Own Evals as a first-class system, turning traces into datasets and graders.
• Develop agentic harnesses and environments for reproducible evaluation.
• Build post-training infrastructure for SFT/RL runs and deployment.
• Create reward signals and verifiers to enable hill-climbing improvements.
• Address latency and cost with efficient model routing and batching.

Skills

Production AI systems

Eval harnesses

Metrics-driven

Model-agnostic thinking

High agency

AI-native tooling

Claude/Cursor/Codex familiarity

Tools

Claude

Cursor

Codex

Job description

NextToken is seeking a Member of Technical Staff in San Francisco to build and scale production AI systems. You will own end-to-end evaluation pipelines, harnesses for agents, post-training workflows, and interfaces that turn experimental results into deployable improvements.

The role emphasizes model-agnostic reasoning, metric-driven development, and hands-on work with leading AI tooling to optimize latency, cost, and quality at scale.