Remote AI Evaluation Analyst (16-Week Contract)

Weekday AI; Comisión de Fomento de Perú; Hybrid; ARS 25,820,000 - 36,774,000; Full time

Job summary

Weekday AI seeks detail-oriented Business Analysts – Non-US Raters to evaluate AI chatbot outputs against predefined criteria. This fully remote, 16-week contract requires 8 hours daily with overlap to US Pacific Time, five days per week, possibly including weekends.

You will review responses, apply evaluation guidelines, and document judgments with accuracy, contributing to improved AI response quality and reliability.

Qualifications

• Bachelor's degree in Business Administration, Finance, Computer Science, or a related field.
• At least 3 years of experience as a Business Analyst or in a similar analytical role.
• Strong analytical thinking, problem-solving, and decision-making abilities.
• Excellent written communication skills and the ability to explain findings clearly.
• Strong attention to detail and the ability to assess information objectively.
• Proficiency in Microsoft Excel, SQL, Power BI, Tableau, or comparable analytical tools.
• Experience with requirement gathering, process mapping, and business documentation.
• Ability to work independently, manage multiple priorities, and meet strict deadlines.
• Strong documentation skills and the ability to follow structured evaluation guidelines.
• Familiarity with Agile or Scrum methodologies is an advantage.
• Willingness to commit to a full-time, 16-week remote engagement with the required time-zone overlap.

Responsibilities

• Review, analyze, and compare responses generated by multiple AI chatbots against predefined evaluation criteria.
• Conduct structured assessments to determine the accuracy, relevance, completeness, clarity, and overall quality of chatbot responses.
• Analyze business requirements and translate them into clear evaluation guidelines and functional specifications.
• Gather, interpret, and document relevant information to support consistent and objective assessments.
• Examine workflows, processes, and data to identify gaps, inconsistencies, and opportunities for improvement.
• Prepare evaluation reports, summaries, and supporting documentation with a strong focus on accuracy.
• Collaborate with project managers and cross-functional teams to clarify requirements and resolve evaluation-related questions.
• Track task progress, quality metrics, and project deliverables while meeting established deadlines.
• Identify recurring response-quality issues and provide actionable observations based on the defined assessment framework.
• Maintain consistency, confidentiality, and high-quality standards throughout the evaluation process.

Skills

Analytical thinking

Problem-solving

Decision-making

Written communication

Attention to detail

Independent work

Multitasking

Documentation

Education

Bachelor's degree in Business Admin/Finance/CS

Tools

Microsoft Excel

SQL

Power BI

Tableau

Job description

Weekday AI seeks detail-oriented Business Analysts – Non-US Raters to evaluate AI chatbot outputs against predefined criteria. This fully remote, 16-week contract requires 8 hours daily with overlap to US Pacific Time, five days per week, possibly including weekends.

You will review responses, apply evaluation guidelines, and document judgments with accuracy, contributing to improved AI response quality and reliability.