Data Science Expert - AI Evaluation

Visa Hunt; Germany (OH); On-site; USD 165,000 - 234,000; Full time

Job summary

Mercor is seeking a Data Science Expert for a contract position. This remote role offers compensation at $120-$170/hour. You will design precise grading criteria for data science deliverables and score AI-generated and human work with detailed justifications.

Apply your expertise in experiment design, A/B testing, and metric definition to help ensure scoring is reproducible and defensible, while incorporating feedback from senior reviewers and working independently to meet deadlines.

Qualifications

• 5+ years of professional data science experience in industry.
• Experience in business operations, product, or growth data science at top-tier tech firms.
• Deep fluency in experiment design, A/B testing, metric definition, and executive communication.
• Excellent written communication.
• Detail-oriented with reproducible judgment.

Responsibilities

• Design precise grading criteria for data science deliverables.
• Score AI-generated and human work with detailed justifications.
• Ensure scoring is reproducible and defensible.
• Incorporate feedback from senior reviewers and iterate.
• Work independently and asynchronously to meet deadlines while improving AI model performance.

Skills

Data science experience

Experiment design & A/B testing

SQL & Python analysis

Executive communication

Attention to detail

Tools

SQL

Python

Job description

About the job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey.

Position: Data Science Expert
Type:Contract
Compensation:$120-$170/hour
Location:Remote

Role Responsibilities

• Design precise, task‑specific grading criteria for real‑world data science deliverables such as analyses, models, dashboards, and experiment readouts.
• Score AI‑generated and human work samples against criteria with detailed written justifications for every score.
• Apply consistent, evidence‑based judgment to ensure scores are reproducible and defensible.
• Incorporate structured feedback from senior reviewers and iterate quickly on your work.
• Work independently and asynchronously to meet deadlines while improving AI model performance.

Qualifications

Must‑Have

• 5+ years of professional data science experience in industry.
• Background in business operations, product, or growth data science at top‑tier technology companies.
• Deep fluency in experiment design and A/B testing, metric definition, SQL/Python analysis, and communicating findings to executive stakeholders.
• Exceptionally strong written communication.
• Detail‑oriented, consistent, and comfortable having your judgment reviewed and calibrated against peers.

Preferred

• Prior experience with AI training, evaluation, or human‑data projects.