Staff AI Systems Engineer: Evaluation & Post-Training Infra

NextToken AI Inc.; San Francisco (CA); On-site; USD 180,000 - 230,000; Full time

Benefits offered by this job

SFT & RLHF tooling

RL environments

Synthetic data pipelines

Inference optimization

Open research / OSS

Job summary

NextToken AI Inc. in San Francisco is seeking an experienced AI systems engineer to lead evaluation and post-training tooling.

You will turn production traces into golden datasets, automated graders, and regression tests, ensuring every change ships with a measurable number. You will build and maintain agentic harnesses and environments for evaluation, data generation, and RL, plus data pipelines for SFT and RL runs.

Qualifications

• 3+ years building production AI systems. You've shipped real systems to real users.
• Numbers over opinions. You've built eval harnesses from scratch and know the difference between a metric that hill-climbs and a metric that lies.
• Model-agnostic. You reason about prompt vs. context vs. model changes and know when each one wins.
• High agency. You find the highest-leverage problem and ship the fix without waiting for a spec.
• AI-native. You use Claude, Cursor, and Codex daily and have earned opinions on where they shine and where they mislead.

Responsibilities

• Evals as a first-class system. Turn production traces into golden datasets, automated graders, and regression tests. Every change ships with a number.
• Agentic harnesses and environments: the tooling that runs agents reproducibly for evaluation, data generation, and RL.
• Post-training infrastructure. Data pipelines, SFT and RL runs, and the plumbing that takes a specialized model from experiment to deployment.
• Reward signals and verifiers. Make fuzzy notions of quality measurable and hill-climbable.
• Latency and cost. Model routing, caching, parallel tool calls, and serving specialized models economically alongside frontier ones.

Skills

Production AI systems

Eval harnesses

Model-agnostic reasoning

High agency

Claude/Cursor/Codex usage

Job description

NextToken AI Inc. in San Francisco is seeking an experienced AI systems engineer to lead evaluation and post-training tooling.

You will turn production traces into golden datasets, automated graders, and regression tests, ensuring every change ships with a measurable number. You will build and maintain agentic harnesses and environments for evaluation, data generation, and RL, plus data pipelines for SFT and RL runs.