Research Engineer

Kindredventures; San Francisco (CA); On-site; USD 175,000 - 250,000; Full time

Benefits offered by this job

In-office perks and daily lunch

Medical, dental, and vision benefits

Unlimited PTO

Equity stake

Job summary

Scorecard is seeking a Research Engineer to build a simuser orchestrator and a scalable simuser framework. You will convert real transcripts into digital twins that reflect user goals, knowledge and quirks, and you will design rigorous evaluations to quantify fidelity.

You will ship the framework so simusers operate inside training environments and contribute to measurable improvements in AI evaluation. You will work with production Python/TypeScript, iterate quickly, and own design decisions

Qualifications

• Proficient in production Python or TypeScript and able to own a system from design through deploy.
• Experience building agentic loops or multi-agent systems with eval in the loop.
• Fidelity-focused: design evaluations with baselines, held-out data, and calibrated judges.

Responsibilities

• Build the simuser orchestrator to manage conversations and context among simusers.
• Build the simuser framework with a structured schema for goals, knowledge, and style.
• Convert transcripts into simusers by turning work context and transcripts into a digital twin.
• Design and run evaluations to quantify fidelity of grounded simulated users.
• Ship the framework to production so simusers run inside training environments.

Skills

Python

TypeScript

LLM fluency

Multi-agent systems

Job description

Research Engineer

About Scorecard

We’re a small, nimble team backed by top-tier investors. We have multi-billion dollar customers and a long runway. To accelerate our growth, we’re looking to hire exceptional engineers to help shape the future of AI development.
This role is part of the "simuser research" pod within Scorecard, and we work closely with our customers to build multi-user simulations.

The role

This team focuses on creating simulations for agents interacting with multiple users, which means we need simulated users (“simusers”) to interact with the agent. Currently, engineers simulate users using vibecoded prompts like "act frustrated," resulting in unrealistic users that corrupt evaluation results.
To fix this, we’re building a simuser framework and orchestration layer. It allows defining high-fidelity simusers and realistically managing a group of simusers that interact with the agent-under-test in a simulation.
We’re hiring a research engineer to help us build and evaluate this framework.

What you'll do

• Build the simuser orchestrator. A simulation containing multiple simusers has to manage conversations and context between simusers. An orchestrator is necessary to prevent simulation drift and keep simusers realistic.
• Build the simuser framework. We need a structured schema to define the simuser’s goals, knowledge, and style, and a tool harness for the simusers to take action in the environment.
• Convert transcripts into simusers. Our simusers start from transcripts of a real human doing the task. You'll build the pipeline that turns work context and transcripts into a digital twin that replicates that person's goals, knowledge, quirks, and memory.
• Design and run evals. We need to validate that our approach to creating simusers results in sufficiently realistic simulations. We plan to publish a benchmark to quantify the fidelity of a grounded simulated user.
• Ship to production. Scorecard's platform consumes everything you build; your simusers run inside training environments where their failures become the agent's habits.

What we’re looking for

• Engineering fundamentals. You write production Python or TypeScript and have owned a system from design through deploy.
• LLM fluency. You've built agentic loops or multi-agent systems and iterated on them with an eval in the loop.
• Empirical rigor. Fidelity claims need evidence. You've designed evaluations with baselines, held-out data, and calibrated judges, and you notice when a metric is easy to hack.
• Comfort with messy data. Our simusers are grounded in real human data. You've worked with data like this and know how much cleaning and judgment it takes before anything structured comes out of it.
• A feel for agent behavior. You can read a simulated conversation and tell when the user is off, like being too articulate or too helpful. You can turn that observation into a rubric or a prompt change, and evaluate it on whether it actually helped.
• Hands-on ownership. The framework is new and the team is small. You're comfortable making design decisions with incomplete information, building the first version yourself, and revising it when the data disagrees with you.

Nice to have

• Published research: papers, benchmarks, datasets, or public experiments.
• Prior work on user simulation: simulated users for dialogue or agent benchmarks, digital twins, role-play systems, or character consistency.
• Evaluation work: LLM-as-judge design, grader calibration, or human annotation pipelines.
• RL experience: building RL environments for LLMs, reward design, or firsthand encounters with reward hacking.
• Behavioral or social-science research methods: survey design, test-retest protocols, inter-rater reliability.

Compensation and benefits

• In-office perks and daily lunch
• Medical, dental, and vision benefits
• Unlimited PTO
• Annual salary range: $175,000 - $250,000 + competitive equity stake