Staff Engineer, AI Runtime & Evaluation Systems

Sycamore; Palo Alto (CA); On-site; USD 170,000 - 260,000; Full time

Benefits offered by this job

Equity

Competitive compensation

Job summary

Sycamore is building the trusted agent operating system for the enterprise. Our small, engineering-led team works directly with Fortune 500 companies to deploy agentic apps with the security and control large organizations need.

You will help build the runtime, evaluation, and learning systems that power the platform, owning end-to-end slices from the agent runtime to environments and self-improvement, collaborating with product, infrastructure, and Forge teams.

Qualifications

• 5-12 years of software engineering experience.
• Strong backend and distributed-systems fundamentals, including typed API design, asynchronous workflows, persistence, reliability, and production debugging.
• Depth in at least one area and genuine interest in working across the others.
• An empirical approach to AI quality: form a hypothesis, design evaluation, interpret data, and distinguish real improvement.
• Ability to reason about retries, idempotency, partial failure, long-running state, concurrency, latency, cost, experiment design, and safe rollout.
• Security-minded approach to multi-tenant systems, identity, credentials, privacy, and auditability.
• Product judgment: durable abstractions behind real requirements.
• AI-native: use coding agents and modern models while owning architecture and outcomes.
• Comfort building product-facing software with React and TypeScript for UI when needed.
• Comfort with startup ambiguity, fast feedback, and broad ownership.

Responsibilities

• Develop the learning data plane around agents: structured trajectories, feedback and outcome signals, offline datasets, lineage, privacy controls, and reliable links between an agent version and its behavior.
• Create evaluation systems for task completion, tool use, long-horizon behavior, safety, latency, and cost, and own the credibility of the numbers they produce.
• Design experiment and versioning systems for comparing changes through replay, shadow traffic, canaries, or controlled rollouts, with clear promotion and rollback criteria.
• Build durable orchestration for long-running tasks, checkpoints, approvals, handoffs, and human-in-the-loop interactions, with typed tool interfaces, protocol-based execution, and memory retrieval that enforces tenant, user, and project visibility boundaries.
• Own the execution environments agents run in, including isolation, startup performance, resource limits, and the policy layer that gates self-modification so a proposed change is staged and tested rather than applied silently.
• Publish reliable APIs, event-driven interfaces, and reusable libraries that work across agent categories and enterprise deployments.

Skills

Backend systems

Distributed systems

Typed API design

Evaluation design

Security-minded

Product judgment

AI-native mindset

React/TypeScript

Startup adaptability

Tools

React

TypeScript

Python

Job description

Sycamore is building the trusted agent operating system for the enterprise. Our small, engineering-led team works directly with Fortune 500 companies to deploy agentic apps with the security and control large organizations need.

You will help build the runtime, evaluation, and learning systems that power the platform, owning end-to-end slices from the agent runtime to environments and self-improvement, collaborating with product, infrastructure, and Forge teams.