AI Evaluation Specialist: Analytic Rigor & Sourcing

Crossing Hurdles; United States; Remote; USD 80,000 - 120,000; Full time

Job summary

Crossing Hurdles in the United States is seeking a seasoned analytics professional to develop and map analytic cases to defined taxonomies across multiple evaluation areas. You will author complex test cases to evaluate model performance on source handling and analytic rigor.

You will assess and grade AI-generated responses based on reasoning, source use, and practical utility, identify over-refusal, document edge cases, and collaborate with project leads to refine procedures for

Qualifications

• Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline.
• Strong proficiency in source evaluation and confidence-level assessment.
• Experience in teaching, mentoring, or leading analytic reviews with a focus on analytic rigor.
• Exceptional written and verbal communication skills with clarity and precision.
• Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
• Familiarity with structured analytic techniques and intelligence tradecraft standards.

Responsibilities

• Develop and map analytic cases to defined taxonomies across multiple evaluation areas.
• Author complex test cases to evaluate model performance on source handling and analytic rigor.
• Assess and grade AI-generated responses based on reasoning, source use, and practical utility.
• Identify and flag instances of over-refusal in analytic exploration by the model.
• Document edge cases and analytic disagreements to support evaluation methodology improvement.
• Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment.
• Contribute insights to calibrate analytic standards and enhance AI system benchmarks.

Skills

Analytic experience

Source evaluation

Teaching/mentoring

Communication skills

Model limitations

Structured analytic techniques

Job description

Crossing Hurdles in the United States is seeking a seasoned analytics professional to develop and map analytic cases to defined taxonomies across multiple evaluation areas. You will author complex test cases to evaluate model performance on source handling and analytic rigor.

You will assess and grade AI-generated responses based on reasoning, source use, and practical utility, identify over-refusal, document edge cases, and collaborate with project leads to refine procedures for