Model Behavior Engineer

Notion; San Francisco (CA); On-site; USD 98,000 - 140,000; Full time

Benefits offered by this job

Competitive salary

Equity options

Comprehensive benefits package

Job summary

A growing technology company located in San Francisco is seeking an innovative Quality Engineer for their AI products. This role blends ops, strategy, and analytics to improve product quality. As part of the AI engineering team, you will use data to shape how AI behaves, work with partners in leading labs, and ensure user satisfaction through effective evaluation baselines. Competitive salary and benefits offered, with a focus on collaboration and quality improvement.

Qualifications

• Proven problem-solving skills, with a bias to action.
• Ability to explore AI product capabilities.
• Experience in handling and analyzing large datasets.

Responsibilities

• Design and iterate on system prompts for AI products.
• Analyze production data to identify and debug issues.
• Develop evaluation strategies for quality tracking.
• Evaluate and launch new AI models with research labs.
• Drive quality narratives while collaborating with teams.
• Build tools to manage AI observability and quality.

Skills

Driver mentality

Curiosity

Analytical instinct

Comfortable working with data

Clear communication

Experience with LLMs, prompting, or AI products

Job description

About Us

Notion helps you build beautiful tools for your life's work. In today's world of endless apps and tabs, Notion provides one place for teams to get everything done, seamlessly connecting docs, notes, projects, calendar, and email—with AI built in to find answers and automate work. Millions of users, from individuals to large organizations like Toyota, Figma, and OpenAI, love Notion for its flexibility and choose it because it helps them save time and money.

In-person collaboration is essential to Notion's culture. We require all team members to work from our offices on Mondays, Tuesdays, and Thursdays, our designated Anchor Days. Certain teams or positions may require additional in‑office workdays.

About The Role

You’ll own the quality bar for Notion AI products. You’ll work with product and engineering teams to build systems to define what “good” looks like, measure our progress, and drive changes to deliver reliable and high‑quality AI experiences. Your work directly shapes how Notion's AI products behave for millions of users.

This isn't a traditional software engineering role. It’s an art & science role. You won't spend your days writing code. Instead, you'll focus on understanding and shaping how our AI products behave through context engineering, designing evaluation systems, and analyzing data. This team sits in our AI engineering team, working directly with engineering, product, design, and data.

This role is a unique blend of ops, strategy, and product thinking. Day to day, you'll live in production data, ship prompt fixes, run evals and, in effect, shape our quality strategy. As part of that you'll shape Notion’s model strategy and work directly with frontier AI labs (OpenAI, Anthropic, Google) to evaluate and launch new models.

We're looking for problem‑seeking generalists interested in 0 → 1: curious people with high agency who thrive in ambiguous, fast‑moving product areas. We're building a product, but also building a new function. You'll have real ownership from day one and help write the playbook as we scale.

What You’ll Achieve

• Context engineering — Design, test, and iterate on system prompts, tool prompts, and context strategies that shape how Notion’s AI products behave. Understand the nuances of how models respond to different context structures and use that knowledge to drive quality improvements directly.
• Understand & debug — Live in production data: transcripts, logs, user feedback. Reproduce issues, identify root causes, and translate symptoms into actionable problem statements. Find signal in noisy data.
• Build evals & Measurement — Design eval strategies, build datasets, run evaluations. Track quality over time. Identify issues before users do. Own the loop: define quality goals, create evals, test and improve.
• Evaluate and launch new models with leading research labs — Evaluate and launch models from OpenAI, Anthropic, Google, and others. Benchmark across dimensions: quality, latency, cost, edge cases. Help shape Notion’s model strategy based on real data.
• Drive quality priorities — Work embedded with eng and product teams to surface the most important issues. Own the quality narrative: severity, frequency, what to fix and why. Be the voice of quality in the room.
• Build tooling & systems — Help manage AI observability and eval platforms (e.g., Braintrust). Build the playbooks and tools that enable all teams at Notion to build AI products.

Skills You’ll Need To Bring

• Driver mentality — You treat problems as yours. If something's broken, it's your job to fix it, even if you didn't cause it. You have a bias to action.
• Curiosity — You’re excited about exploring the “jagged frontier” of LLM capabilities and how AI products meet reality.
• Analytical instinct — Your first move is to look at data. You can find signal in noise.
• Comfortable working with data — You can self‑serve insights from large datasets, whether through SQL, coding agents, or other tools.
• Clear communication — You can explain complex issues simply.
• Experience with LLMs, prompting, or AI products

Nice To Have's

• Backgrounds in engineering, product, data science, research, consulting
• You've built something on your own to solve a problem — side project, startup, tool, whatever

Equal Opportunity Employment

Notion is proud to be an equal opportunity employer. We do not discriminate in hiring or any employment decision based on race, color, religion, national origin, age, sex (including pregnancy, childbirth, or related medical conditions), marital status, ancestry, physical or mental disability, genetic information, veteran status, gender identity or expression, sexual orientation, or other applicable legally protected characteristic. Notion considers qualified applicants with criminal histories, consistent with applicable federal, state and local law. Notion is also committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability, please let your recruiter know.

Compensation and Benefits

Notion is committed to providing highly competitive cash compensation, equity, and benefits. The compensation offered for this role will be based on multiple factors such as location, the role’s scope and complexity, and the candidate’s experience and expertise, and may vary from the range provided below. For roles based in San Francisco or New York City, the estimated base salary range for this role is $98,000 - $140,000 per year.

Application Process

By clicking "Submit Application", I understand and agree that Notion and its affiliates and subsidiaries will collect and process my information in accordance with Notion's Global Recruiting Privacy Policy and NYLL 144.