AI Evaluation Intelligence Analyst - Remote

YO AI Labs

New York (NY)

Remote

USD 69,000 - 96,000

Part time

1 hour ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

YO AI Labs is seeking an experienced Intelligence Analyst to contribute to evaluating and improving next-generation AI systems. You will apply intelligence analysis skills to assess AI outputs, evaluate reasoning, and help improve accuracy, reliability, and consistency of AI models.

Remote contractor role requiring strong analytical judgment, clear communication, and the ability to document edge cases and provide actionable feedback to the project leads for continuous improvement.

Qualifications

  • All-source intelligence analysis experience or civilian IC expertise.
  • Strong source evaluation, confidence-level assessment, and established intelligence analysis standards.
  • Experience teaching, mentoring, or leading analytical reviews.
  • Excellent written and verbal communication with clarity, precision, and evidence-based reasoning.
  • Ability to identify and explain nuanced model limitations, including cases of unnecessary refusal.
  • Familiarity with structured analytical techniques and intelligence tradecraft standards.
  • Previous experience in AI evaluation, model assessment, or related quality assurance initiatives is a plus.

Responsibilities

  • Develop and map analytical cases to taxonomies across evaluation areas.
  • Create complex test cases to evaluate model performance in source evaluation and reasoning.
  • Assess and grade AI-generated responses based on reasoning quality and source use.
  • Document edge cases and ambiguous scenarios to support continuous improvement.
  • Collaborate with project leads to refine procedures for source evaluation and confidence-level assessment.
  • Provide insights to calibrate analytical standards and improve AI evaluation benchmarks.

Skills

Intelligence analysis
Source evaluation
Analytical rigor
Communication
Model limitations
Structured analytic techniques
AI evaluation

Job description

YO AI Labs is seeking an experienced Intelligence Analyst to contribute to evaluating and improving next-generation AI systems. You will apply intelligence analysis skills to assess AI outputs, evaluate reasoning, and help improve accuracy, reliability, and consistency of AI models.

Remote contractor role requiring strong analytical judgment, clear communication, and the ability to document edge cases and provide actionable feedback to the project leads for continuous improvement.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote Senior Intelligence Analyst – Contractor
Remote Senior Intelligence Analyst – Contractor

YO AI Labs • New York (NY)

Remote
USD 62,000 - 117,000
Intelligence Analyst - Remote
Intelligence Analyst - Remote

YO AI Labs • New York (NY)

Remote
USD 69,000 - 96,000
Intelligence Analyst
Intelligence Analyst

confidential • United States

Remote
USD 55,000 - 96,000
AI Evaluation Data Analyst - Remote Contractor
AI Evaluation Data Analyst - Remote Contractor

Remotebridge • United States

Remote
USD 60,000 - 90,000
Remote Evaluation Specialist - AI Model Testing & Research
Remote Evaluation Specialist - AI Model Testing & Research

NEPSE Trading • United States

Remote
USD 28,000 - 62,000
Remote Junior Data Scientist - AI Evaluation and Feedback
Remote Junior Data Scientist - AI Evaluation and Feedback

YO AI Labs • New York (NY)

Remote
USD 42,000 - 78,000
Remote Junior Data Scientist: AI Evaluation & Feedback
Remote Junior Data Scientist: AI Evaluation & Feedback

YO AI Labs • New York (NY)

Remote
USD 60,000 - 90,000
Intelligence Analyst remote
Intelligence Analyst remote

Executive Staff Recruiters / ESR Healthcare • United States

Remote
USD 55,000 - 96,000
AI Generalist Contractor — Remote, Real-World Tasks
AI Generalist Contractor — Remote, Real-World Tasks

YO AI Labs • New York (NY)

Remote
USD 34,000 - 55,000
Remote Intelligence Analyst for AI Evaluation and Analytics
Remote Intelligence Analyst for AI Evaluation and Analytics

Micro1 • United States

Remote
USD 70,000 - 110,000
Refer-a-friend bonus