On-Device AI Evaluation Architect

Rnb Consultancy; San Francisco (CA); On-site; USD 200,000 - 250,000; Full time

Benefits offered by this job

Equity

Relocation support

Visa/immigration assistance

Job summary

San Francisco AI Lab is hiring to build and own evaluation ecosystems for on-device agents and model releases. You will define readiness criteria, design human-scored rubrics, and ship dashboards and tooling to speed research and leadership decisions.

Ideal candidates excel at engineering instrumentation, rapid experiment cycles, and communicating what constitutes meaningful improvement under deadline pressure.

Qualifications

• Design and maintain evaluation harnesses for AI models and agents.
• Create robust rubrics and ensure evaluation quality under tight deadlines.
• Build dashboards and tooling to accelerate research iterations and leadership decisions.

Responsibilities

• Own the eval suites for model and agent releases (capabilities, behavior, regressions).
• Define what "ready to ship" means with concrete evidence.
• Bridge research, product engineering, and partnerships to translate improvements into verifiable commitments.

Skills

Eval harnesses

Dashboards

Instrumentation

Rubrics design

On-device evaluation

Tools

Dashboards tooling

Experiment tooling

Job description

San Francisco AI Lab is hiring to build and own evaluation ecosystems for on-device agents and model releases. You will define readiness criteria, design human-scored rubrics, and ship dashboards and tooling to speed research and leadership decisions.

Ideal candidates excel at engineering instrumentation, rapid experiment cycles, and communicating what constitutes meaningful improvement under deadline pressure.