AI Output Evaluator: 26-03373

Akraya Inc.; San Francisco (CA); Remote; USD 62,000 - 69,000; Part time

Job summary

AKRAYA is seeking a part-time remote data and content evaluator to meticulously review AI-generated data narratives for accuracy and clarity. You will assess charts, headlines, and conversational outputs against a defined quality rubric, identifying subtle errors that automated systems might miss.

The role emphasizes strong analytical skills, editorial judgment, and precise written communication to help improve AI-driven data storytelling.

Qualifications

• Strong analytical skills for data interpretation.
• Excellent attention to detail for editorial review.
• Proficiency in clear and concise written communication.

Responsibilities

• Evaluate AI-generated content against a specific quality rubric.
• Assess the accuracy and appropriateness of data visualizations and claims.
• Identify and report misleading information or awkward phrasing in AI output.
• Document evaluation findings and provide concise reports to the team.
• Propose enhancements to the evaluation rubric based on observed trends.

Skills

Data analysis

Data visualization

Editorial judgment

Written reporting

LLM evaluation

Job description

Primary Skills

• Data analysis (advanced)
• Data visualization (advanced)
• Editorial judgment (advanced)
• Written reporting (advanced)
• LLM evaluation (intermediate)

Contract Type

W2

Duration

1+ Months (Part-time 15 hours/week)

Location

Remote

Pay Range

$45 - $50 per hour on W2

Job Summary

This role involves meticulously reviewing AI-generated data narratives to ensure accuracy and clarity. You will assess charts, headlines, and conversational responses against established criteria, identifying subtle errors or misinterpretations that automated systems might miss. Your feedback will be crucial in refining the AI's ability to produce honest and effective data storytelling.

Key Responsibilities

• Evaluate AI-generated content against a specific quality rubric.
• Assess the accuracy and appropriateness of data visualizations and claims.
• Identify and report misleading information or awkward phrasing in AI output.
• Document evaluation findings and provide concise reports to the team.
• Propose enhancements to the evaluation rubric based on observed trends.

Must-Have Skills

• Strong analytical skills for data interpretation.
• Excellent attention to detail for editorial review.
• Proficiency in clear and concise written communication.

Domain / Industry Experience

Experience in data analytics or content evaluation, particularly with AI-generated content, is preferred.

About AKRAYA

Akraya is an award-winning IT staffing firm consistently recognized for our commitment to excellence and a thriving work environment. Most recently, we were recognized Stevie Employer of the Year 2025, SIA Best Staffing Firm to work for 2025, Inc 5000 Best Workspaces in US (2025 & 2024) and Glassdoor's Best Places to Work (2023 & 2022)! Industry Leaders in Tech Staffing As Talent solutions provider for Fortune 100 Organizations, Akraya's industry recognitions solidify our leadership position in the IT staffing space. We don't just connect you with great jobs, we connect you with a workplace that inspires!