Member of Technical Staff, Agents

Wwshemi; Palo Alto (CA); Hybrid; USD 200,000 - 300,000; Full time

Job summary

Wwshemi in Palo Alto is hiring to build autonomous AI assistants for executive workflows. You will develop agent capabilities and evaluation systems to ensure real-world tasks are completed reliably, with measurable quality.

The role emphasizes end-to-end agent development, designing evaluation frameworks, and shipping production-grade ML systems. It offers a high annual salary and an on-site four days a week in Palo Alto with a flexible remote component.

Qualifications

• 3+ years of data science or ML experience building evaluation systems, metrics, or quality measurement for products.
• Production-quality Python and SQL skills, including experience analyzing large datasets and building automated pipelines.
• Experience designing evaluation frameworks, datasets, measurable criteria, and appropriate metrics.
• A track record delivering production AI, machine learning, agent, or agent-evaluation systems for real users.
• Experience improving agent reliability, owning systems end to end, and measuring results in production.

Responsibilities

• Build agents that complete delegated work end to end for real users.
• Define reliable measures of success for agent tasks and maintain representative evaluations.
• Develop new agent capabilities and validate them before they reach users.
• Investigate production failures and deliver fixes that address underlying causes.
• Build systems to detect known failure patterns and emerging issues at scale.
• Improve agent recovery when tools or websites fail.
• Create repeatable methods for engineers to measure the impact of agent changes.

Skills

Data science

Machine learning

Evaluation metrics

Python

SQL

Experiment design

Statistics

Production AI

End-to-end systems

Shipping in production

Tools

Django

React

TypeScript

Braintrust

Raindrop

Job description

This listing's locations Remote - Palo Alto

Employment Full-time

Workplace Remote

Compensation USD 200000–300000 per year

Join a small, senior team building autonomous AI assistants for executive workflows. You will develop agent capabilities and evaluation systems that help ensure real-world tasks are completed reliably, with measurable quality.

What You'll Do

• Build agents that complete delegated work end to end for real users.
• Define reliable measures of success for agent tasks and maintain representative evaluations.
• Develop new agent capabilities and validate them before they reach users.
• Investigate production failures and deliver fixes that address underlying causes.
• Build systems to detect known failure patterns and emerging issues at scale.
• Improve agent recovery when tools or websites fail.
• Create repeatable methods for engineers to measure the impact of agent changes.

What We're Looking For

• At least 3 years of data science or machine learning experience building evaluation systems, metrics, or quality measurement for products.
• Production-quality Python and SQL skills, including experience analyzing large datasets and building automated pipelines.
• Experience designing evaluation frameworks, datasets, measurable criteria, and appropriate metrics.
• A track record delivering production AI, machine learning, agent, or agent-evaluation systems for real users.
• Experience improving agent reliability, owning systems end to end, and measuring results in production.
• Familiarity with statistics, experiment design, and realistic end-to-end simulations; experience with fine-tuning, post-training, or reinforcement learning is valuable.
• Experience with Python, Django, React, and TypeScript; familiarity with evaluation tools such as Braintrust or Raindrop is useful.
• Demonstrated progression and sustained experience in a role, with the ability to work through ambiguity and ship quickly.

Base salary is $200,000 to $300,000 USD annually. Visa sponsorship is not available.

Location

On-site in Palo Alto, California, with four days a week in the office.