Intelligence Analyst remote

Executive Staff Recruiters / ESR Healthcare; United States; Remote; USD 55,000 - 96,000; Part time

Job summary

Executive Staff Recruiters / ESR Healthcare seeks an Intelligence Analyst for a remote contractor role. You will map analytic cases, author test scenarios, and assess AI outputs across source handling, reasoning, and uncertainty.

Strong communication and analytic rigor are essential to advance evaluation methodologies. The position emphasizes all-source analysis, structured techniques, and the ability to articulate model limitations.

Qualifications

• All-source analytic experience or deep expertise in a civilian-IC style discipline.
• Strong ability to evaluate sources and assign confidence levels.
• Proven analytic rigor and ability to articulate limitations of AI outputs.
• Experience teaching, mentoring, or leading analytic reviews.
• Familiarity with structured analytic techniques and intelligence tradecraft standards.

Responsibilities

• Develop and map analytic cases to defined taxonomies across multiple evaluation areas.
• Author complex test cases to evaluate model performance on source handling, analytic rigor, and uncertainty.
• Assess and grade AI-generated responses based on reasoning, sourcing, limitations, and usefulness.
• Identify and flag over-refusal instances and document feedback for improvement.
• Document edge cases and analytic disagreements to support method development.
• Collaborate with leads to refine evaluation procedures for source evaluation and confidence levels.
• Contribute insights to calibrate analytic standards and improve AI benchmarks quality.

Skills

All-source analytic experience

Source evaluation expertise

Confidence-level assessment

Analytic rigor

Ability to articulate model limits

Structured analytic techniques

AI evaluation participation

Strong written and verbal comms

Job description

Intelligence Analyst remote

Role Type: Contractor

Location: Remote

Scope of Work

1. Develop and map analytic cases to defined taxonomies across multiple evaluation areas, ensuring comprehensive coverage of relevant intelligence scenarios.
2. Author complex test cases designed to evaluate model performance on source handling, analytic rigor, and nuanced uncertainty expression.
3. Assess and grade AI-generated responses based on criteria such as reasoning, source use, recognition of limitations, and practical utility.
4. Identify and flag instances where legitimate analytic exploration is over-refused by the model, providing clear documentation and feedback.
5. Document edge cases and analytic disagreements, supporting continuous improvement of evaluation methodologies.
6. Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment.
7. Contribute insights to help calibrate analytic standards and enhance the overall quality of AI system benchmarks.

Preferred Qualifications

1. Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline, ideally with a civilian-IC background.
2. Strong proficiency in source evaluation, confidence-level assessment, and established analytic standards.
3. Experience in teaching, mentoring, or leading analytic reviews, with a record of fostering analytic rigor.
4. Exceptional written and verbal communication skills, with a focus on clarity and precision.
5. Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
6. Familiarity with structured analytic techniques and intelligence tradecraft standards.
7. Previous participation in AI evaluation or model assessment initiatives is a plus.