Remote AI Evaluation Specialist — Domain Expert

Zekai; United States; Remote; USD 34,000 - 96,000; Part time

Job summary

Zekai is seeking a Domain Contributor - Evaluation Specialist to support an internal project evaluating and improving advanced AI models. This remote contractor role focuses on designing domain-specific tasks, assessing outputs against primary sources, and documenting clear rationales.

You will work across history, law, finance, journalism, and tech, with flexible, pay-per-task compensation and expectations to submit tasks weekly. Start immediately for rapid onboarding.

Qualifications

• Proven hands‑on experience with AI model training, annotation, RLHF, or evaluation.
• Deep expertise in a core domain: history, archival or library science, law (paralegal, legal academic, compliance), finance (equity research, PE/VC, forensic accounting), journalism, or technology/industry history.
• Advanced research skills with a track record of verifying complex claims against authoritative sources.
• Exceptional written and verbal communication skills to articulate findings and feedback.
• Demonstrated ability to precisely follow detailed instructions and evaluation frameworks.
• Background in adversarial or stomp the model task design is advantageous.
• Graduate‑level research, experience at a Fortune 100 company, or professional fluency in a non‑English language a plus.

Responsibilities

• Design nuanced, domain-specific tasks that probe AI model limits and surface reasoning flaws.
• Author questions and scenarios demanding deep subject knowledge, beyond general understanding.
• Assess AI-generated outputs, validating accuracy against authoritative sources in your area of expertise.
• Document clear rationales for model shortcomings, specifying missed reasoning, source errors, or nuances.
• Apply detailed task guidelines and evaluation rubrics consistently to each submission.
• Submit completed deliverables independently on a rolling basis, meeting project requirements for written and verbal clarity.

Skills

AI model training

Annotation

RLHF

Evaluation

Job description

Zekai is seeking a Domain Contributor - Evaluation Specialist to support an internal project evaluating and improving advanced AI models. This remote contractor role focuses on designing domain-specific tasks, assessing outputs against primary sources, and documenting clear rationales.

You will work across history, law, finance, journalism, and tech, with flexible, pay-per-task compensation and expectations to submit tasks weekly. Start immediately for rapid onboarding.