Remote Intelligence Analyst - AI Benchmark Evaluator

Executive Staff Recruiters / ESR Healthcare; United States; Remote; USD 55,000 - 96,000; Part time

Job summary

Executive Staff Recruiters / ESR Healthcare seeks an Intelligence Analyst for a remote contractor role. You will map analytic cases, author test scenarios, and assess AI outputs across source handling, reasoning, and uncertainty.

Strong communication and analytic rigor are essential to advance evaluation methodologies. The position emphasizes all-source analysis, structured techniques, and the ability to articulate model limitations.

Qualifications

• All-source analytic experience or deep expertise in a civilian-IC style discipline.
• Strong ability to evaluate sources and assign confidence levels.
• Proven analytic rigor and ability to articulate limitations of AI outputs.
• Experience teaching, mentoring, or leading analytic reviews.
• Familiarity with structured analytic techniques and intelligence tradecraft standards.

Responsibilities

• Develop and map analytic cases to defined taxonomies across multiple evaluation areas.
• Author complex test cases to evaluate model performance on source handling, analytic rigor, and uncertainty.
• Assess and grade AI-generated responses based on reasoning, sourcing, limitations, and usefulness.
• Identify and flag over-refusal instances and document feedback for improvement.
• Document edge cases and analytic disagreements to support method development.
• Collaborate with leads to refine evaluation procedures for source evaluation and confidence levels.
• Contribute insights to calibrate analytic standards and improve AI benchmarks quality.

Skills

All-source analytic experience

Source evaluation expertise

Confidence-level assessment

Analytic rigor

Ability to articulate model limits

Structured analytic techniques

AI evaluation participation

Strong written and verbal comms

Job description

Executive Staff Recruiters / ESR Healthcare seeks an Intelligence Analyst for a remote contractor role. You will map analytic cases, author test scenarios, and assess AI outputs across source handling, reasoning, and uncertainty.

Strong communication and analytic rigor are essential to advance evaluation methodologies. The position emphasizes all-source analysis, structured techniques, and the ability to articulate model limitations.