Applied ML Evaluation Specialist (Remote Contract)

OpenTrain AI; Northern (KY); Hybrid; USD 96,000 - 124,000; Part time

Job summary

OpenTrain AI is recruiting an Applied Machine Learning Task Auditor to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train frontier AI models. The work centers on evaluation rather than development.

You will determine whether experiments are designed soundly, model-selection reasoning is supported by evidence, and evaluation methods produce trustworthy conclusions.

Qualifications

• Hands-on applied or experimental machine-learning experience (3+ years).
• Experience with experiment design, model selection, and hyperparameter tuning.
• Strong data-quality controls and data leakage detection.
• Knowledge of train/test and cross-validation hygiene.
• Proficiency with PyTorch, TensorFlow, scikit-learn, and XGBoost.

Responsibilities

• Review applied machine-learning tasks for methodological rigor and evidentiary support.
• Assess hyperparameter-tuning choices and evaluation methodology.
• Check for data leakage, metric gaming, and train/test hygiene issues.
• Provide clear written feedback using structured rubrics.
• Reproduce results when necessary.

Job description

OpenTrain AI is recruiting an Applied Machine Learning Task Auditor to evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train frontier AI models. The work centers on evaluation rather than development.

You will determine whether experiments are designed soundly, model-selection reasoning is supported by evidence, and evaluation methods produce trustworthy conclusions.