GenAI Evaluation Scientist (Remote, 35h/wk)

Mercor; New York (NY); Remote; USD 90,000 - 120,000; Full time

Job summary

Cincinnatus LLC is seeking experienced machine learning practitioners to act as ground-truth experts for model evaluation and experimentation on a leading AI lab's GenAI team. You will author complex, multi-step ML tasks and verify where frontier models fall short.

This full-time W-2 role is fully remote in the United States, about 35 hours per week, with the opportunity to be placed at a top AI lab as part of the extended workforce.

Qualifications

• MSc or PhD in ML, CS, or related STEM field, or equivalent research experience.
• 1+ years in a research or research-engineering role.
• Hands-on experience training and evaluating ML models end-to-end.
• Strong familiarity with large language models and evaluation techniques.
• Proficiency in Python and Git; comfortable with scripting and notebooks.
• Basic understanding of reinforcement learning is preferred.
• Past experience in AI training, model evaluation, or benchmark/task authoring is preferred.
• Attention to detail, creativity, and ability to work independently in ambiguous problems.
• Available to commit ~35 hours per week.

Responsibilities

• Design tasks: Turn real ML research ideas into well-defined, multi-step tasks.
• Run experiments: Implement changes, run training experiments, and analyze results.
• Explore RL ideas: Build tasks around reinforcement-learning basics.
• Evaluate models: See how frontier models handle tasks and note shortcomings.
• Work as a team: Compare notes with researchers to ensure consistency and rigor.

Skills

Python

Git

LLMs

Research experience

Education

MSc or PhD in ML/CS

Job description

Cincinnatus LLC is seeking experienced machine learning practitioners to act as ground-truth experts for model evaluation and experimentation on a leading AI lab's GenAI team. You will author complex, multi-step ML tasks and verify where frontier models fall short.

This full-time W-2 role is fully remote in the United States, about 35 hours per week, with the opportunity to be placed at a top AI lab as part of the extended workforce.