AI Policy Evaluator — Precise, Contextual Classifications (Onsite)

Apply; Seattle (WA); On-site; USD 62,000 - 76,000; Full time

Job summary

Handshake is seeking an AI Policy Generalist in Seattle to translate complex customer policies into consistent, well-reasoned evaluations of AI model behavior. You will read user requests, model outputs, and history to determine the correct policy category and cite evidence from the text and context.

You will balance policy guidance with customer expectations, contribute to calibration discussions, and write precise rationales.

Responsibilities

• Learn new customer policies, definitions, taxonomies, and evaluation rubrics quickly
• Evaluate user requests and AI model responses within the full relevant conversation context
• Distinguish between closely related labels, severity levels, and policy boundaries
• Select the most defensible classification when a case is genuinely ambiguous
• Write concise, evidence-based rationales citing relevant policy language and conversation details
• Identify policy gaps, contradictions, unclear definitions, and emerging edge cases
• Raise thoughtful questions when existing guidance does not resolve a case
• Participate actively in calibration discussions with evaluators, project leads, policy teams, and researchers
• Challenge interpretations respectfully and update your judgment when new guidance or stronger reasoning emerges
• Apply customer policy consistently without substituting personal beliefs for the policy standard
• Maintain accuracy and attention to detail across repeated evaluations
• Incorporate feedback quickly and apply clarified guidance to future work
• Help improve evaluation frameworks, examples, decision rules, and quality standards
• Move effectively between projects covering different policy domains and customer needs

Job description

Handshake is seeking an AI Policy Generalist in Seattle to translate complex customer policies into consistent, well-reasoned evaluations of AI model behavior. You will read user requests, model outputs, and history to determine the correct policy category and cite evidence from the text and context.

You will balance policy guidance with customer expectations, contribute to calibration discussions, and write precise rationales.