AI Model Behavior Engineer—Quality & Evaluation

Notion; San Francisco (CA); On-site; USD 98,000 - 140,000; Full time

Benefits offered by this job

Competitive salary

Equity options

Comprehensive benefits package

Job summary

A growing technology company located in San Francisco is seeking an innovative Quality Engineer for their AI products. This role blends ops, strategy, and analytics to improve product quality. As part of the AI engineering team, you will use data to shape how AI behaves, work with partners in leading labs, and ensure user satisfaction through effective evaluation baselines. Competitive salary and benefits offered, with a focus on collaboration and quality improvement.

Qualifications

• Proven problem-solving skills, with a bias to action.
• Ability to explore AI product capabilities.
• Experience in handling and analyzing large datasets.

Responsibilities

• Design and iterate on system prompts for AI products.
• Analyze production data to identify and debug issues.
• Develop evaluation strategies for quality tracking.
• Evaluate and launch new AI models with research labs.
• Drive quality narratives while collaborating with teams.
• Build tools to manage AI observability and quality.

Skills

Driver mentality

Curiosity

Analytical instinct

Comfortable working with data

Clear communication

Experience with LLMs, prompting, or AI products

Job description

A growing technology company located in San Francisco is seeking an innovative Quality Engineer for their AI products. This role blends ops, strategy, and analytics to improve product quality. As part of the AI engineering team, you will use data to shape how AI behaves, work with partners in leading labs, and ensure user satisfaction through effective evaluation baselines. Competitive salary and benefits offered, with a focus on collaboration and quality improvement.