Applied ML Evaluator & AI Rigor Reviewer

Obsidian; Philadelphia (Philadelphia County); On-site; USD 120,000 - 170,000; Full time

Job summary

Obsidian seeks a candidate to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate frontier AI lab models. You will critique experiment design, model-selection reasoning, and evaluation methodology, and provide rubric-based feedback to improve rigor.

The role emphasizes hands-on analysis of data quality and reproducibility, with a focus on leakage detection and proper train/test hygiene, using PyTorch, TensorFlow, scikit-learn,

Qualifications

• 3+ years hands-on applied/experimental ML experience.
• Strong grasp of data-quality rigor: leakage detection, metric gaming, and train/test/CV hygiene.
• Proficiency with standard ML frameworks: PyTorch, TensorFlow, scikit-learn, XGBoost.
• Ability to critique ML claims against evidence and reproduce results.

Responsibilities

• Evaluate the quality, correctness, and methodological rigor of applied ML tasks.
• Assess experiment design, model-selection reasoning, and evaluation methodology.
• Provide rubric-based written feedback to guide improvements.

Skills

Applied ML

Experiment design

Model selection reasoning

Evaluation methodology

Data quality rigor

ML frameworks

Tools

PyTorch

TensorFlow

scikit-learn

XGBoost

Job description

Obsidian seeks a candidate to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate frontier AI lab models. You will critique experiment design, model-selection reasoning, and evaluation methodology, and provide rubric-based feedback to improve rigor.

The role emphasizes hands-on analysis of data quality and reproducibility, with a focus on leakage detection and proper train/test hygiene, using PyTorch, TensorFlow, scikit-learn,