AI Policy Evaluator & Case Reasoning Specialist

Cacheflow; Seattle (WA); On-site; USD 48,000 - 138,000; Full time

Job summary

Handshake is seeking an AI Policy Generalist in Seattle, WA to turn complex customer policies into consistent evaluations of AI model behavior. You will read user requests, model outputs, and history to determine the best policy category, handling cases where a single word or detail changes the classification.

This is a precision-focused role built for careful reasoning and calibration with the team. The position emphasizes clear explanations, evidence-based rationales, and ongoing learning to

Qualifications

• Strong candidates may come from quality assurance, research, editing, law, teaching, operations, trust and safety, content moderation, or other fields that require careful interpretation and defensible decision-making.
• You enjoy making precise distinctions between cases that others might consider equivalent.
• You can explain judgment calls clearly enough that another person can audit your reasoning.

Responsibilities

• Learn new customer policies, definitions, taxonomies, and evaluation rubrics quickly.
• Evaluate user requests and AI model responses within the full relevant conversation context.
• Distinguish between closely related labels, severity levels, and policy boundaries.
• Select the most defensible classification when a case is genuinely ambiguous.
• Write concise, evidence-based rationales that cite relevant policy language and conversation details.
• Identify policy gaps, contradictions, unclear definitions, and emerging edge cases.
• Raise thoughtful questions when existing guidance does not resolve a case.
• Participate actively in calibration discussions with evaluators, project leads, policy teams, and researchers.
• Challenge interpretations respectfully and update your judgment when new guidance or stronger reasoning emerges.
• Apply customer policy consistently without substituting personal beliefs for the policy standard.
• Maintain accuracy and attention to detail across repeated evaluations.
• Incorporate feedback quickly and apply clarified guidance to future work.
• Help improve evaluation frameworks, examples, decision rules, and quality standards.
• Move effectively between projects covering different policy domains and customer needs.

Job description

Handshake is seeking an AI Policy Generalist in Seattle, WA to turn complex customer policies into consistent evaluations of AI model behavior. You will read user requests, model outputs, and history to determine the best policy category, handling cases where a single word or detail changes the classification.

This is a precision-focused role built for careful reasoning and calibration with the team. The position emphasizes clear explanations, evidence-based rationales, and ongoing learning to