Founding ML Research Engineer

Pally (YC S25); San Francisco (CA); On-site; USD 200,000 - 300,000; Full time

Benefits offered by this job

Health insurance

401(k)

Job summary

Pally is hiring for a senior to staff-level role in San Francisco, focused on hands-on ML research and engineering. You will work in the eval harness, training code, and the agent itself, building end-to-end evaluation, fine-tuning, and routing systems to improve the assistant over time.

You’ll contribute to defenses against prompt injection, design robust evaluation data, and shape the product’s memory and tool-use strategies, with a strong emphasis on shipping reliable models.

Qualifications

• 4+ years of ML or software engineering, with experience at an AI lab, AI product company, or similar.
• Hands-on experience post-training or fine-tuning LLMs (SFT, preference tuning, distillation) and shipping results to production.
• Experience designing evals, such as LLM judges, human preference data, or replay of real traffic, trusted to make ship decisions.

Responsibilities

• Evaluations: Build the harness that converts feedback into a score for each reply.
• Fine-tuning: Turn conversations into training data and distill models for production.
• Model routing: Train a router so small models handle most turns and larger models handle crucial ones.
• Red teaming: Identify prompts and other attack vectors; build defenses and evals.
• Agent experience: Integrate memory, retrieval, tool selection, and decide when to act or ask.

Skills

Python

TypeScript

ML engineering

Evaluations design

Post-training distillation

Router/LLM system design

Job description

Hey , we’re Haz and Wyatt, the founders of Pally.

The role

This is a hands-on research and engineering role. You’ll be in the eval harness, the training code, and the agent itself. In practice that means:

• Evaluations. Build the harness that turns “did that reply feel right?” into a number: turn-level judges, human preference data, replay of real conversations, and regression suites that run before every deploy. Evals the team trusts to make ship decisions.
• Fine-tuning. Turn real conversations into training and preference data, and post-train and distill models (SFT, preference tuning, distillation) so small, fast models do what frontier models do well.
• Model routing. Train and evaluate a router so small models take most turns and frontier models take the ones that matter. Own the latency, cost, and quality tradeoff per turn.
• Red teaming. Find the ways Pally can be tricked, leak something, or take an action nobody asked for, from prompt injection in an email to a malicious web page. Build the defenses and the evals that keep them working.
• Agent experience. Bring ML across the agent: memory and retrieval, tool selection, when to act versus ask, and when to reach out first. Read transcripts every day and turn what you see into evals and model changes.

We’ll measure you on one thing: how much better Pally gets at being someone’s assistant, week over week, and whether the evals prove it.

Profile

This is a senior to staff level role. We’re looking for someone who has already done this work at a frontier AI lab or a team operating at that level.

• 4+ years of ML or software engineering, with experience at an AI lab, AI product company, or similar.
• Hands-on experience post-training or fine-tuning LLMs (SFT, preference tuning, distillation) and shipping the results to production.
• Experience designing evals, such as LLM judges, human preference data, or replay of real traffic, that a team trusted to make ship decisions.
• Strong Python and the ML stack for training and evals, and comfortable working in TypeScript in the product codebase.
• Fluency with AI-assisted development workflows, grounded in the fundamentals to verify, correct, and own everything you ship.
• You want founding-team scope. You’ll set the research bar for Pally and help hire the engineers who join after you.

Practical stuff

$200-300k base + significant equity, 401(k), and health insurance. Full-time, in person in San Francisco, CA.

We work exceptionally hard. If work isn’t your primary focus, this isn’t for you. However, we also give you full autonomy over your schedule. We don’t care when you work hard, only that you do.