Member of Technical Staff, Agents

Royper Royco LLC

Palo Alto (CA)

On-site

USD 200,000 - 300,000

Full time

31 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Royper Royco LLC in Palo Alto seeks a data scientist or ML engineer to join a small, senior team building autonomous AI agents for executive workflows.

You will develop evaluation systems, metrics, and end-to-end agent capabilities, ensuring reliable task completion with measurable improvements for users.

Base salary: $200,000 to $300,000 annually. Visa sponsorship is not available.

Qualifications

  • 3+ years of data science or ML experience building evaluation systems, metrics, or product quality measurement.
  • Production-quality Python and SQL skills for analysis, large datasets, and automated pipelines.
  • Experience designing evaluation frameworks, representative datasets, meaningful metrics, and realistic end-to-end simulations.
  • Track record delivering production AI, ML, agent, or agent-evaluation systems for real users and measuring their impact.
  • Experience improving agent reliability, owning systems end to end, and using statistics and experiment design to evaluate changes.
  • Familiarity with post-training, fine-tuning, or reinforcement learning is useful.
  • Experience with Django, React, and TypeScript; the role also uses Python, SQL, Braintrust, and Raindrop.
  • Demonstrated progression and sustained experience in a role, with comfort working in a fast-moving, ambiguous environment.

Responsibilities

  • Build agents that complete delegated work end to end, without requiring users to check each step.
  • Define measurable success criteria and trusted ways to assess agent performance across tasks.
  • Develop and evaluate new agent capabilities before releasing them to users.
  • Investigate production failures and deliver durable fixes based on their underlying causes.
  • Build systems to detect known failure patterns and less predictable issues at scale, using tools such as Braintrust and Raindrop.
  • Improve agent recovery when tools or websites fail.
  • Create repeatable evaluation workflows that let engineers measure whether changes improve agent performance.

Skills

Data science / ML eval systems
Python
SQL
Experiment design
Agent reliability
Production AI/ML
Statistics
RL familiarity

Tools

Django
React
TypeScript
Braintrust
Raindrop

Job description

Join a small, senior team building autonomous AI agents for executive workflows. You will develop agent capabilities and evaluation systems that help ensure real-world tasks are completed reliably, with measurable improvements reaching users quickly.

What You'll Do
  • Build agents that complete delegated work end to end, without requiring users to check each step.
  • Define measurable success criteria and trusted ways to assess agent performance across tasks.
  • Develop and evaluate new agent capabilities before releasing them to users.
  • Investigate production failures and deliver durable fixes based on their underlying causes.
  • Build systems to detect known failure patterns and less predictable issues at scale, using tools such as Braintrust and Raindrop.
  • Improve agent recovery when tools or websites fail.
  • Create repeatable evaluation workflows that let engineers measure whether changes improve agent performance.
What We're Looking For
  • At least 3 years of data science or machine learning experience building evaluation systems, metrics, or product quality measurement.
  • Production-quality Python and SQL skills for analysis, large datasets, and automated pipelines.
  • Experience designing evaluation frameworks, representative datasets, meaningful metrics, and realistic end-to-end simulations.
  • A track record delivering production AI, ML, agent, or agent-evaluation systems for real users and measuring their impact.
  • Experience improving agent reliability, owning systems end to end, and using statistics and experiment design to evaluate changes.
  • Familiarity with post-training, fine-tuning, or reinforcement learning is useful.
  • Experience with Django, React, and TypeScript; the role also uses Python, SQL, Braintrust, and Raindrop.
  • Demonstrated progression and sustained experience in a role, with comfort working in a fast-moving, ambiguous environment.

Base salary: $200,000 to $300,000 annually. Visa sponsorship is not available.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Member of Technical Staff, Agents
Member of Technical Staff, Agents

Clera • Palo Alto (CA)

On-site
USD 200,000 - 300,000
Senior Agent Engineer
Senior Agent Engineer

Job Envoy • Palo Alto (CA)

On-site
USD 180,000 - 240,000
Senior Staff Engineer - Autonomous AI Agents (Production)
Senior Staff Engineer - Autonomous AI Agents (Production)

Clera • Palo Alto (CA)

On-site
USD 200,000 - 300,000
AI Agent Engineer
AI Agent Engineer

Staging Meetgail • Miami (FL)

Hybrid
USD 180,000 - 250,000
Member of Technical Staff, AI Agents
Member of Technical Staff, AI Agents

True Talent Solutions Partners • New York (NY)

On-site
USD 150,000 - 300,000
Visa sponsorship
Applied AI Engineer
Applied AI Engineer

daydream Labs • San Francisco (CA)

On-site
USD 180,000 - 220,000
Medical, dental, and vision insurance
Lunch on in-office days
Wellness and learning stipends
+2
Member of Technical Staff, Product
Member of Technical Staff, Product

Raydar • San Francisco (CA)

On-site
USD 200,000 - 400,000
Equity
Visa sponsorships
Senior Engineer, AI/ML
Senior Engineer, AI/ML

TriSearch • Denver (CO)

Hybrid
USD 160,000 - 190,000
Health coverage
Flexible work options (Hybrid in US)
401(k) match
+5
Staff AI Engineer, Agents and Evals
Staff AI Engineer, Agents and Evals

MimoLabs • United States

Remote
USD 170,000 - 220,000
Equity with four-year vesting and one‑
Staff Applied AI Engineer, Agents
Staff Applied AI Engineer, Agents

Morena • Boston (MA)

Hybrid
USD 180,000 - 240,000
Equity
Company benefits
Hybrid work