AI Evaluation Data Scientist - Remote Contract

Mercor; United States; On-site; USD 137,760 - 206,640; Part time

Job summary

Mercor is seeking a Data Scientist Talent Network contractor to design grading criteria for data science deliverables and evaluate AI-generated or human-created work against established criteria. You will provide written justifications to ensure transparency and accountability while calibrating judgments with senior reviewers.

The role requires 1+ year of data science experience, strong Python/SQL skills, and the ability to work independently and asynchronously in a remote setting.

Qualifications

• 1+ years of professional data science experience.
• Experience at a leading technology, research, or quantitative firm.
• Strong command of Python, SQL, statistical modeling, machine learning, and experimentation.
• Exceptional written communication skills.
• Detail-oriented and consistent in evaluating complex work.
• Comfort receiving feedback and calibrating judgment against established standards.

Responsibilities

• Design precise, task-specific grading criteria for data science deliverables, including EDA, modeling, ML pipelines, and AB test write-ups.
• Evaluate AI-generated or human-created work against established criteria to ensure quality and consistency.
• Provide detailed written justifications for evaluations and scores to maintain transparency and accountability.
• Apply consistent, evidence-based judgment to ensure assessments are reproducible and defensible.
• Incorporate structured feedback from senior reviewers and iterate on submitted work for continuous improvement.
• Work independently and asynchronously to meet deadlines and improve AI model performance.

Skills

Python

SQL

Statistical Modeling

Machine Learning

Experimentation

Written Communication

Detail-oriented

Feedback Calibration

Tools

Python

SQL

Statistical Modeling

Job description

Mercor is seeking a Data Scientist Talent Network contractor to design grading criteria for data science deliverables and evaluate AI-generated or human-created work against established criteria. You will provide written justifications to ensure transparency and accountability while calibrating judgments with senior reviewers.

The role requires 1+ year of data science experience, strong Python/SQL skills, and the ability to work independently and asynchronously in a remote setting.