Intelligence Analyst

Confidential; United States; Remote; USD 55,000 - 96,000; Part time

Job summary

Zekai is seeking an Intelligence Analyst for a 100% remote contract role. You will contribute to evaluating and advancing AI capabilities, training models through high-quality real-world input, and shaping how systems learn and reason.

Experience in AI evaluation is helpful, but domain knowledge matters most. As a contractor, you will develop analytic cases, grade AI responses, and document edge cases while collaborating with project leads to refine evaluation procedures and standards on a

Qualifications

• Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline is preferred.
• Strong ability to evaluate sources and assign confidence levels to analyses.
• Experience teaching or leading analytic reviews is a plus.
• Excellent written and spoken communication with precision and clarity.
• Ability to articulate model limitations and over-refusal scenarios clearly.

Responsibilities

• Develop and map analytic cases to defined taxonomies across evaluation areas.
• Author complex test cases to evaluate AI model performance on source handling and uncertainty.
• Assess and grade AI-generated responses for reasoning and practicality.
• Identify when analytic exploration is over-refused by the model and document feedback.
• Document edge cases and analytic disagreements to improve methodologies.
• Collaborate with project leads to refine evaluation procedures and confidence scoring.
• Contribute insights to calibrate analytic standards and overall benchmark quality.

Skills

All-source analysis

Analytic rigor

Source evaluation

Mentoring/lead analytic reviews

Structured analytic techniques

Education

Civilian-IC background

Job description

Role Title: Intelligence Analyst

Role Type: Contractor

Location: Remote

our partner is engaging Intelligence Analysts to contribute to a customer-facing project focused on evaluating and advancing artificial intelligence capabilities. In this role, you'll apply your expertise to help train next-generation AI systems. Your work will shape how models learn, reason, and perform through high-quality, real-world input. No prior experience in AI is required — your domain knowledge is what matters.

Scope of Work

• Develop and map analytic cases to defined taxonomies across multiple evaluation areas, ensuring comprehensive coverage of relevant intelligence scenarios.
• Author complex test cases designed to evaluate model performance on source handling, analytic rigor, and nuanced uncertainty expression.
• Assess and grade AI-generated responses based on criteria such as reasoning, source use, recognition of limitations, and practical utility.
• Identify and flag instances where legitimate analytic exploration is over-refused by the model, providing clear documentation and feedback.
• Document edge cases and analytic disagreements, supporting continuous improvement of evaluation methodologies.
• Collaborate with project leads to refine procedures for source evaluation and confidence-level assignment.
• Contribute insights to help calibrate analytic standards and enhance the overall quality of AI system benchmarks.

Preferred Qualifications

• Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline, ideally with a civilian-IC background.
• Strong proficiency in source evaluation, confidence-level assessment, and established analytic standards.
• Experience in teaching, mentoring, or leading analytic reviews, with a record of fostering analytic rigor.
• Exceptional written and verbal communication skills, with a focus on clarity and precision.
• Ability to identify and articulate nuanced model limitations and over-refusal scenarios.
• Familiarity with structured analytic techniques and intelligence tradecraft standards.
• Previous participation in AI evaluation or model assessment initiatives is a plus.

This is a 100% remote contract role, open to applicants worldwide in the countries the employer accepts.