AI Quality & Evaluation Architect

Intermedia Lab; San Francisco (CA); On-site; USD 98,000 - 140,000; Full time

Job summary

Notion is seeking a role focused on quality for Notion AI products. You will shape evaluation systems, analyze data, and collaborate with engineering, product, design, and data teams to ensure reliable AI experiences.

This role blends operations, strategy, and product thinking. You will work with frontier labs and own the quality narrative while driving improvements across models and prompts.

Qualifications

• Experience with LLMs, prompting, or AI products.
• Driver mentality with bias to action.
• Curiosity about frontier AI capabilities and AI product reality.
• Analytical instinct: base decisions on data and signals.
• Clear communication of complex issues to cross-functional teams.
• Ability to self-serve insights from large datasets or tools.

Responsibilities

• Own the quality bar for Notion AI products.
• Design evals, collect data, and drive improvements.
• Work with eng, product, design, and data to shape model strategy.
• Collaborate with frontier AI labs to evaluate new models.
• Build tooling for AI observability and evaluation platforms.
• Translate user feedback into actionable quality goals.

Skills

LLM prompting

AI product experience

Data analysis

Problem solving

Clear communication

Curiosity

Driver mentality

Team collaboration

Job description

Notion is seeking a role focused on quality for Notion AI products. You will shape evaluation systems, analyze data, and collaborate with engineering, product, design, and data teams to ensure reliable AI experiences.

This role blends operations, strategy, and product thinking. You will work with frontier labs and own the quality narrative while driving improvements across models and prompts.