Applied ML Task Auditor & Rubric Evaluator

Obsidian; San Francisco (CA); On-site; USD 120,000 - 210,000; Full time

Job summary

Obsidian is seeking an evaluation-focused ML scientist to assess applied ML rigor in frontier AI research settings. You will critique experiment designs, model-selection reasoning, and evaluation methodologies, delivering rubric-based feedback to improve reproducibility and validity.

Responsibilities emphasize identifying data leakage, metric gaming, and CV hygiene issues, ensuring claims are well-supported by evidence and experiments are replicable across datasets and baselines.

Qualifications

• 3+ years hands-on applied ML, including experiment design and evaluation.
• Strong data-quality rigor: leakage detection, metric gaming, train/test/CV hygiene.
• Proficiency with standard ML frameworks: PyTorch, TensorFlow, scikit-learn, XGBoost.
• Ability to critique ML claims against evidence and reproduce results.

Responsibilities

• Evaluate the quality, correctness, and rigor of applied ML tasks used to train and evaluate frontier AI models.
• Assess experiment design, model-selection reasoning, and evaluation methodology.
• Provide rubric-based written feedback to improve reproducibility and validity.

Skills

Applied ML

Experiment design

Model evaluation

Tools

PyTorch

TensorFlow

scikit-learn

XGBoost

Job description

Obsidian is seeking an evaluation-focused ML scientist to assess applied ML rigor in frontier AI research settings. You will critique experiment designs, model-selection reasoning, and evaluation methodologies, delivering rubric-based feedback to improve reproducibility and validity.

Responsibilities emphasize identifying data leakage, metric gaming, and CV hygiene issues, ensuring claims are well-supported by evidence and experiments are replicable across datasets and baselines.