A.I Policy Reviewer

Odixcity Consulting; La Réunion; Sur place; EUR 78 815 - 105 087; Plein temps

Résumé du poste

Odixcity Consulting is seeking an AI Policy Reviewer to evaluate both AI-generated and user content for compliance with internal governance and responsible AI principles. You will assess outputs for safety, bias, and policy alignment, serving as a crucial quality gate before deployment.

The role requires strong policy knowledge, clear written communication, and the ability to navigate complex edge cases with nuanced judgement. Remote, international scope.

Qualifications

• Minimum 3 years in Trust & Safety, policy, risk, or compliance.
• Deep understanding of content moderation and harmful content policies.
• Ability to deconstruct AI responses and identify biases.
• Strong written communication to explain AI missteps to engineers.
• Experience with dashboards and review tools; familiarity with LLMs.
• Resilience handling disturbing AI content.
• Attention to global cultural nuances.

Responsabilités

• Review AI and user content against policy rubrics.
• Act as QA checkpoint for automated systems.
• Handle edge cases with nuanced judgement.
• Identify systematic flaws and report biases.
• Collaborate with Policy to refine rubrics.
• Participate in red-teaming to test resilience.
• Explain ratings to ML engineers.
• Create prompts and golden examples for training.

Connaissances

Trust & Safety

Policy Analysis

Content Moderation

Communication Skills

LLM Familiarity

Outils

ChatGPT

Gemini

Description du poste

Job Title

AI Policy Reviewer

Location

Remote (Worldwide)

Job Summary

The AI Policy Reviewer is responsible for evaluating AI-generated and user-generated content to ensure compliance with internal governance standards, regulatory requirements, and responsible AI principles. This role plays a key part in safeguarding model integrity by reviewing outputs for safety risks, bias, misinformation, harmful content, and policy violations, while ensuring consistent enforcement of AI usage guidelines.

Responsibilities

• Review and score AI-generated responses against detailed policy rubrics. Assess outputs for safety, truthfulness, fairness, and alignment with community guidelines.
• Act as a quality assurance checkpoint for automated systems. Identify instances where the AI misinterprets policy (e.g., being over-sensitive and censoring benign content, or under-sensitive and allowing harmful content).
• Handle complex “edge cases” where policy application is ambiguous. Make nuanced judgement calls regarding context, satire, or emerging risks that the AI model struggles to process.
• Analyze and review data to identify systematic flaws in the AI’s reasoning. Report patterns of bias, hallucination, or policy gaps to the Product and Engineering teams.
• Collaborate with Policy teams to test and refine evaluation rubrics. Provide feedback on whether current policies are “teachable” to AI models or if they require human-only judgement.
• Participate in adversarial testing (red teaming) by attempting to “jailbreak” the model or provoke unsafe responses to identify vulnerabilities before launch.
• Work closely with Machine Learning Engineers to explain the “why” behind your ratings, helping them adjust model behavior.
• Write high-quality examples (prompts and ideal responses) that serve as “golden sets” for training the AI on how to handle difficult policy scenarios.

Requirements

• Minimum of 3 years of professional experience in Trust & Safety Operations, Content Policy, Risk Analysis, or Legal/Compliance review.
• Deep understanding of content moderation principles, including hate speech, harassment, misinformation, and graphic violence policies.
• Strong ability to deconstruct complex AI responses and identify logical flaws, hallucinations, or subtle biases.
• Clear and concise written communication skills. You must be able to explain why an AI response was wrong in a way that engineers and policy experts can understand.
• This role involves exposure to disturbing AI-generated text and images designed to test safe limits. Proven emotional resilience and self-care strategies are required.
• Comfortable working with dashboards, spreadsheets, and specialized review tools. Familiarity with LLMs (ChatGPT, Gemini, etc.).
• Proven ability to follow complex, detailed instructions and scoring rubrics with high consistency and accuracy.
• Understanding of global cultural and political nuances to assess whether AI responses are appropriate for diverse international audiences.