ML Evaluation Engineer - Experiment Design & Data Quality

Anyone AI Inc.; United States; Remote; USD 96,000 - 152,000; Part time

Benefits offered by this job

Remote project-based engagement

Job summary

anyone-ai is seeking specialists to review and evaluate ML challenges used in AI model training and evaluation. The role focuses on debugging experiments and designing rigorous evaluations to avoid shortcuts and artifact reliance.

Ideal candidates have hands-on ML experience, strong design and evaluation skills, and the ability to ensure reproducible, well-calibrated tasks across CPU/GPU workflows in a remote, project-based setting.

Qualifications

• 3+ years of hands-on applied machine learning experience.
• Strong experience with experiment design, model selection, hyperparameter tuning, and model evaluation.
• Deep understanding of train, validation, and test splits and sound preprocessing practice.
• Ability to identify data leakage, label noise, distribution shift, and contamination.
• Familiarity with statistical significance, effect sizes, and choosing appropriate ML evaluation metrics.
• Experience debugging ML workloads across CPU and GPU environments.

Responsibilities

• Review ML challenges to determine whether they are well designed and technically solvable.
• Evaluate whether datasets contain meaningful, learnable signals or rely on artifacts and shortcuts.
• Detect metric gaming, data leakage, label noise, distribution shift, and other evaluation flaws.
• Identify unintended shortcuts or spurious correlations in synthetic datasets.
• Verify reproducibility across the complete data -> model -> evaluation pipeline.
• Assess whether challenge difficulty is appropriately calibrated and provide clear recommendations to recalibrate or exclude problematic tasks.

Job description

anyone-ai is seeking specialists to review and evaluate ML challenges used in AI model training and evaluation. The role focuses on debugging experiments and designing rigorous evaluations to avoid shortcuts and artifact reliance.

Ideal candidates have hands-on ML experience, strong design and evaluation skills, and the ability to ensure reproducible, well-calibrated tasks across CPU/GPU workflows in a remote, project-based setting.