Applied ML Evaluator & Methodology Reviewer

Mercor; Philadelphia (Philadelphia County); On-site; USD 120,000 - 180,000; Full time

Job summary

Mercor is seeking a researcher to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate frontier AI models. The role involves assessing experiment design, model-selection reasoning, and evaluation methodology and providing rubric-based written feedback.

The candidate should demonstrate 3+ years of hands-on applied/experimental ML work, strong data-quality rigor, and proficiency with PyTorch, TensorFlow, scikit-learn, and XGBoost,

Qualifications

• 3+ years hands-on applied/experimental ML (design, selection, tuning, evaluation)
• Strong data-quality rigor: leakage detection, metric gaming, train/test/CV hygiene
• Proficient with standard ML frameworks (PyTorch, TensorFlow, scikit-learn, XGBoost)
• Ability to critique ML claims against evidence and reproduce results

Responsibilities

• Evaluate the quality, correctness, and methodological rigor of applied ML tasks used to train and evaluate models
• Assess experiment design, model-selection reasoning, and evaluation methodology
• Provide clear, rubric-based written feedback

Skills

Applied ML experience

Experiment design

Model selection

Hyperparameter tuning

Evaluation methodology

Data-quality rigor

Framework proficiency

Tools

PyTorch

TensorFlow

scikit-learn

XGBoost

Job description

Mercor is seeking a researcher to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate frontier AI models. The role involves assessing experiment design, model-selection reasoning, and evaluation methodology and providing rubric-based written feedback.

The candidate should demonstrate 3+ years of hands-on applied/experimental ML work, strong data-quality rigor, and proficiency with PyTorch, TensorFlow, scikit-learn, and XGBoost,