Applied ML Evaluation Auditor

Obsidian; San Francisco (CA); Remote; USD 90,000 - 130,000; Full time

Job summary

Obsidian is seeking an applied ML evaluator to assess the quality, correctness, and rigor of experiments used to train frontier AI models. You will review experiment design, model-selection reasoning, and evaluation methodology.

This role emphasizes data-quality hygiene, leakage detection, and reproducibility—no MLOps duties. Experience with PyTorch, TensorFlow, scikit-learn, and XGBoost is expected; prior peer review or benchmarking is a plus.

Qualifications

• 3+ years hands-on applied ML focusing on experiment design, model selection, tuning, and evaluation.
• Strong data-quality discipline: leakage detection, metric gaming, and proper train/test/CV hygiene.
• Proficient with PyTorch, TensorFlow, scikit-learn, and XGBoost.
• Ability to critique ML claims against evidence and reproduce results.

Skills

Experiment design

Model selection

Hyperparameter tuning

Evaluation methodology

Tools

PyTorch

TensorFlow

scikit-learn

XGBoost

Job description

Obsidian is seeking an applied ML evaluator to assess the quality, correctness, and rigor of experiments used to train frontier AI models. You will review experiment design, model-selection reasoning, and evaluation methodology.

This role emphasizes data-quality hygiene, leakage detection, and reproducibility—no MLOps duties. Experience with PyTorch, TensorFlow, scikit-learn, and XGBoost is expected; prior peer review or benchmarking is a plus.