ML Rigor Auditor for Experimental Tasks

Mercor; San Francisco (CA); On-site; USD 130,000 - 190,000; Full time

Job summary

Mercor is seeking a reviewer to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate frontier AI models. You will assess experiment design, model-selection reasoning, and evaluation methodology, and deliver rubric-based written feedback.

The role emphasizes careful data-quality practices, critique of ML claims against evidence, and the ability to reproduce results across experiments.

Qualifications

• 3+ years hands-on applied/experimental ML (design, selection, tuning, evaluation)
• Strong grasp of data-quality rigor: leakage detection, metric gaming, and train/test/CV hygiene
• Proficiency with standard ML frameworks (PyTorch, TensorFlow, scikit-learn, XGBoost)
• Ability to critique ML claims against evidence and reproduce results

Responsibilities

• Evaluate the quality, correctness, and methodological rigor of applied ML tasks used to train and evaluate models at the frontier AI lab.
• Assess experiment design, model-selection reasoning, and evaluation methodology, and provide rubric-based written feedback.
• Identify gaps, propose concrete improvements, and communicate findings clearly to technical and non-technical stakeholders

Job description

Mercor is seeking a reviewer to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate frontier AI models. You will assess experiment design, model-selection reasoning, and evaluation methodology, and deliver rubric-based written feedback.

The role emphasizes careful data-quality practices, critique of ML claims against evidence, and the ability to reproduce results across experiments.