AI Evaluation Specialist: ML Task Auditor

HumanitApp; United States; Hybrid; USD 96,000 - 124,000; Full time

Job summary

HumanitApp is seeking an evaluator to review the quality and rigor of applied ML tasks used to train and assess frontier AI models. You will examine experiment design, model-selection reasoning, and evaluation methodology, delivering clear rubric-based feedback to support research teams.

The role requires 3+ years in AI evaluation or related fields, with emphasis on rigorous methodological critique and precise written communication. Expect a 40-hour work week at a competitive hourly rate.

Qualifications

• 3+ years of experience in AI evaluation or related field.
• Ability to assess experiment designs and evaluation methodologies.
• Strong written communication for rubric-based feedback.

Responsibilities

• Evaluate quality, correctness, and methodological rigor of applied ML tasks used to train and evaluate models.
• Assess experiment design, model-selection reasoning, and evaluation methodology.
• Provide rubric-based written feedback to inform improvements and decision-making.

Skills

AI evaluation

Machine learning

Research

Job description

Future projects, not immediate vacancies. Check each network's country and experience requirements.

Evaluate the quality, correctness, and methodological rigor of applied machine-learning tasks used to train and evaluate a frontier AI lab's models. You'll assess experiment design, model-selection reasoning, and evaluation methodology

and provide clear, rubric-based written feedback.

Basic Qualifications • 3+ years hands-on applied/experimental ML (experiment design, model selection, hyperparameter tuning, evaluation methodology) • Strong grasp of data-quality rigor: leakage detection, metric gaming, and train/test/CV hygiene • Proficiency with standard ML frameworks (PyTorch, TensorFlow, scikit-learn, XGBoost) • Ability to critique ML claims against evidence and reproduce results

Preferred Qualifications • Competition / benchmark experience (e.g., Kaggle) • Graduate research or publication record in applied ML • Prior task-grading or peer-review experience

Note: this role evaluates applied/experimental ML rigor

it is not an LLM-application-building or MLOps role.

Mercor eligibility fields: Eligible work locations: United States.

AI evaluation Machine learning Research

Compensation context

The listing states $70 - $90 / hour. Confirm the final rate, workload, and payment terms during the official application process.

• Proficiency with standard ML frameworks (PyTorch, TensorFlow, scikit-learn, XGBoost)
• Ability to critique ML claims against evidence and reproduce results

Not the right fit?

Use your CV to find relevant roles on HumanitApp. This is separate from a partner application.

Optional role alerts from HumanitApp. Subscribe only if you want weekly emails.

Can I start immediately through an Instant Work Offer?

This listing is not a promise of immediate work. Mercor may send separate offers to prequalified candidates. Read how Instant Work Offers work .

How should I compare the listed pay?

The listing states $70 - $90 / hour. Confirm the final rate, workload, and payment terms during the official application process.

Does Apply use a referral link?

Yes. The button opens the exact verified Mercorlisting using the referral URL published for this role.

Is HumanitApp the employer?

No. HumanitApp independently curates the opportunity. The partner platform manages applications and hiring decisions.

Must I share my details here?

No. Choose the primary Apply action to continue directly without giving HumanitApp your name or email address.