AI Evaluator | $80 Remote

Crossing Hurdles; United States; À distance; USD 80 000 - 120 000; Plein temps

Résumé du poste

Crossing Hurdles in the United States is seeking a seasoned analytics professional to develop and map analytic cases to defined taxonomies across multiple evaluation areas. You will author complex test cases to evaluate model performance on source handling and analytic rigor.

You will assess and grade AI-generated responses based on reasoning, source use, and practical utility, identify over-refusal, document edge cases, and collaborate with project leads to refine procedures for

Qualifications

• Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline.
• Strong proficiency in source evaluation and confidence-level assessment.
• Experience in teaching, mentoring, or leading analytic reviews with a focus on analytic rigor.
• Exceptional written and verbal communication skills with clarity and precision.
• Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
• Familiarity with structured analytic techniques and intelligence tradecraft standards.

Responsabilités

• Develop and map analytic cases to defined taxonomies across multiple evaluation areas.
• Author complex test cases to evaluate model performance on source handling and analytic rigor.
• Assess and grade AI-generated responses based on reasoning, source use, and practical utility.
• Identify and flag instances of over-refusal in analytic exploration by the model.
• Document edge cases and analytic disagreements to support evaluation methodology improvement.
• Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment.
• Contribute insights to calibrate analytic standards and enhance AI system benchmarks.

Connaissances

Analytic experience

Source evaluation

Teaching/mentoring

Communication skills

Model limitations

Structured analytic techniques

Description du poste

• Develop and map analytic cases to defined taxonomies across multiple evaluation areas.
• Author complex test cases to evaluate model performance on source handling and analytic rigor.
• Assess and grade AI-generated responses based on reasoning, source use, and practical utility.
• Identify and flag instances of over-refusal in analytic exploration by the model.
• Document edge cases and analytic disagreements to support evaluation methodology improvement.
• Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment.
• Contribute insights to calibrate analytic standards and enhance AI system benchmarks.

Requirements

• Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline.
• Strong proficiency in source evaluation and confidence-level assessment.
• Experience in teaching, mentoring, or leading analytic reviews with a focus on analytic rigor.
• Exceptional written and verbal communication skills with clarity and precision.
• Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
• Familiarity with structured analytic techniques and intelligence tradecraft standards.