ML Challenge Auditor: Rubric-Based Feedback

Mercor; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

Mercor seeks an ML evaluation specialist to assess applied/experimental rigor in frontier AI lab models. You will critique experiment design, model selection reasoning, and evaluation methodology, and provide rubric-based feedback to researchers.

This role emphasizes data-quality hygiene, reproducibility, and clear, evidence-based judgments. It is not about building or deploying LLMs or MLOps, but about improving scientific rigor through structured reviews and constructive guidance.

Qualifications

• 3+ years hands-on applied/experimental ML (experiment design, model selection, hyperparameter tuning, evaluation methodology)
• Strong grasp of data-quality rigor: leakage detection, metric gaming, and train/test/CV hygiene
• Proficiency with standard ML frameworks (PyTorch, TensorFlow, scikit-learn, XGBoost)
• Ability to critique ML claims against evidence and reproduce results

Responsibilities

• Assess experiment design, model selection reasoning, and evaluation methodology.
• Provide rubric-based written feedback to researchers.
• Ensure documented, reproducible evaluation practices and reporting.

Skills

Applied ML

Experiment design

Model selection

Hyperparameter tuning

Evaluation methodology

Data-quality rigor

Leakage detection

Train/test/CV hygiene

PyTorch

TensorFlow

scikit-learn

XGBoost

Reproduce results

Tools

PyTorch

TensorFlow

scikit-learn

XGBoost

Job description

Mercor seeks an ML evaluation specialist to assess applied/experimental rigor in frontier AI lab models. You will critique experiment design, model selection reasoning, and evaluation methodology, and provide rubric-based feedback to researchers.

This role emphasizes data-quality hygiene, reproducibility, and clear, evidence-based judgments. It is not about building or deploying LLMs or MLOps, but about improving scientific rigor through structured reviews and constructive guidance.