Founding ML Research Engineer - Shape AI Assistants at Scale

Pally (YC S25); San Francisco (CA); On-site; USD 200,000 - 300,000; Full time

Benefits offered by this job

Health insurance

401(k)

Job summary

Pally is hiring for a senior to staff-level role in San Francisco, focused on hands-on ML research and engineering. You will work in the eval harness, training code, and the agent itself, building end-to-end evaluation, fine-tuning, and routing systems to improve the assistant over time.

You’ll contribute to defenses against prompt injection, design robust evaluation data, and shape the product’s memory and tool-use strategies, with a strong emphasis on shipping reliable models.

Qualifications

• 4+ years of ML or software engineering, with experience at an AI lab, AI product company, or similar.
• Hands-on experience post-training or fine-tuning LLMs (SFT, preference tuning, distillation) and shipping results to production.
• Experience designing evals, such as LLM judges, human preference data, or replay of real traffic, trusted to make ship decisions.

Responsibilities

• Evaluations: Build the harness that converts feedback into a score for each reply.
• Fine-tuning: Turn conversations into training data and distill models for production.
• Model routing: Train a router so small models handle most turns and larger models handle crucial ones.
• Red teaming: Identify prompts and other attack vectors; build defenses and evals.
• Agent experience: Integrate memory, retrieval, tool selection, and decide when to act or ask.

Skills

Python

TypeScript

ML engineering

Evaluations design

Post-training distillation

Router/LLM system design

Job description

Pally is hiring for a senior to staff-level role in San Francisco, focused on hands-on ML research and engineering. You will work in the eval harness, training code, and the agent itself, building end-to-end evaluation, fine-tuning, and routing systems to improve the assistant over time.

You’ll contribute to defenses against prompt injection, design robust evaluation data, and shape the product’s memory and tool-use strategies, with a strong emphasis on shipping reliable models.