AI Quality Evaluator for CS Research (Remote)

prolificacademicltd; United States; Remote; USD 68,000 - 97,000; Full time

Benefits offered by this job

Flexible remote participation

Pay-per-task payments up to $60 per hr

Job summary

Prolific Academic Ltd is seeking experts to evaluate AI systems through paid, project-based tasks. You will compare AI responses to technical prompts, review research, and assess code or diagrams for correctness, with an initial skills assessment required for joining the expert pool.

Responsibilities include ranking responses, verifying research summaries, and flagging inaccuracies. A bachelor’s degree in CS is required, and PayPal is used for payments.

Qualifications

• Bachelor’s degree in computer science or a closely related technical field.
• Ability to read research papers and evaluate algorithms and code logic.
• Strong analytical judgment and attention to technical inaccuracies.

Responsibilities

• Compare AI responses to technical prompts for accuracy, reasoning, and safety, then rank them using evaluation guidance.
• Review computer science research alongside generated summaries and visual abstracts for faithful representation of the findings.
• Check technical data, formulas, algorithms, and research claims, flagging errors or unsupported conclusions.
• Assess code snippets and architecture diagrams for logical, structural, and functional correctness.
• Provide expert judgments that help align models with professional expectations in software engineering and data science.

Skills

Analytical judgment

Attention to technical inaccuracies

Code evaluation

Education

Bachelor’s degree in Computer Science

Job description

Prolific Academic Ltd is seeking experts to evaluate AI systems through paid, project-based tasks. You will compare AI responses to technical prompts, review research, and assess code or diagrams for correctness, with an initial skills assessment required for joining the expert pool.

Responsibilities include ranking responses, verifying research summaries, and flagging inaccuracies. A bachelor’s degree in CS is required, and PayPal is used for payments.