AI Evaluation Specialist: ML Task Auditor

HumanitApp; Northern (KY); Hybrid; USD 96,000 - 124,000; Full time

Job summary

HumanitApp is seeking an evaluator to review the quality and rigor of applied ML tasks used to train and assess frontier AI models. You will examine experiment design, model-selection reasoning, and evaluation methodology, delivering clear rubric-based feedback to support research teams.

The role requires 3+ years in AI evaluation or related fields, with emphasis on rigorous methodological critique and precise written communication. Expect a 40-hour work week at a competitive hourly rate.

Qualifications

• 3+ years of experience in AI evaluation or related field.
• Ability to assess experiment designs and evaluation methodologies.
• Strong written communication for rubric-based feedback.

Responsibilities

• Evaluate quality, correctness, and methodological rigor of applied ML tasks used to train and evaluate models.
• Assess experiment design, model-selection reasoning, and evaluation methodology.
• Provide rubric-based written feedback to inform improvements and decision-making.

Skills

AI evaluation

Machine learning

Research

Job description

HumanitApp is seeking an evaluator to review the quality and rigor of applied ML tasks used to train and assess frontier AI models. You will examine experiment design, model-selection reasoning, and evaluation methodology, delivering clear rubric-based feedback to support research teams.

The role requires 3+ years in AI evaluation or related fields, with emphasis on rigorous methodological critique and precise written communication. Expect a 40-hour work week at a competitive hourly rate.