Machine Learning Engineer — Model Evaluation & Experimentation

OpenTrain AI; United States; On-site; USD 82,656 - 123,984; Part time

Job summary

OpenTrain AI is seeking an experienced machine learning practitioner to act as a ground-truth expert for frontier model evaluation. You will turn research ideas into reproducible evaluation tasks, run training experiments, and analyze results to show correct solutions and where models fall short.

This is a contract, part-time position for US-based contributors, expected at 20+ hours per week.

Qualifications

• MSc/PhD in ML, CS or related STEM, or equivalent experience.
• At least 1 year in research or research-engineering role.
• Hands-on experience training and evaluating ML models end-to-end.
• Strong familiarity with large language models, their capabilities, limitations, and evaluation techniques.
• Proficient in Python and Git; comfortable in scripting and notebook environments.

Responsibilities

• Design, implement, and run end-to-end evaluation and experimentation workflows that benchmark language models and training approaches.
• Turn vague ML research ideas into well-defined, multi-step evaluation tasks and benchmarks.
• Implement changes, run training experiments, and analyze outcomes to demonstrate correct behavior.
• Build tasks and analyses around reinforcement learning concepts such as reward functions and training behavior.
• Evaluate frontier models on your tasks, document failure modes, and explain where and why they fall short.
• Collaborate with researchers and other experts to keep tasks consistent, rigorous, and reproducible.

Skills

Research problem solving

Written communication

Independent work

Experiment design

Education

MSc or PhD in ML/CS or related STEM (or equivalent experience)

1+ years in research or research-engineering role

Tools

Python

Git

Job description

About OpenTrain

OpenTrain is the #1 platform for finding and building careers in AI training and data labeling. We help people discover specialized projects, build a unified AI training portfolio, and grow a durable freelance career in an industry that shapes how modern AI systems learn.

About AI training and this work

AI training (data annotation, human feedback, and model evaluation) is the human side of creating reliable AI. Contributors do everything from designing benchmarks to rating outputs and running experiments that teach models what correct behavior looks like.

This role sits at the cutting edge: you will author and run multi-step evaluation experiments that directly inform how next-generation language models are measured and improved.

The role

OpenTrain AI is hiring an experienced machine learning practitioner to act as a ground-truth expert for frontier model evaluation. You will turn research ideas into reproducible evaluation tasks, run training experiments, and analyze results to show correct solutions and where models fall short.

This is a contract, part-time position for US-based contributors, expected at 20+ hours per week.

What you'll do

You will design, implement, and run end-to-end evaluation and experimentation workflows that benchmark language models and training approaches.

• Turn vague ML research ideas into well-defined, multi-step evaluation tasks and benchmarks.
• Implement changes, run training experiments, and analyze outcomes to demonstrate correct behavior.
• Build tasks and analyses around reinforcement learning concepts such as reward functions and training behavior.
• Evaluate frontier models on your tasks, document failure modes, and explain where and why they fall short.
• Collaborate with researchers and other experts to keep tasks consistent, rigorous, and reproducible.

Requirements

You must meet the core qualifications and be able to work independently on open-ended research problems.

• MSc or PhD in machine learning, computer science, or a related STEM field (or equivalent experience).
• 1+ years in a research or research-engineering role.
• Hands-on experience training and evaluating ML models end-to-end, including experiment setup, execution, and analysis.
• Strong familiarity with large language models, their capabilities, limitations, and evaluation techniques.
• Working proficiency in Python and Git, comfortable in both scripting and notebook environments.
• High attention to detail, creativity in task design, and strong written communication skills.
• Ability to work independently on ambiguous, open-ended problems.
• Must be legally able to work in the United States.

Helpful background and tools

The following are not required but will help you move faster in the role.

• Basic understanding of reinforcement learning (reward functions, policy training).
• Past experience in AI training, model evaluation, or benchmark/task authoring.
• Familiarity with evaluation workflows for text tasks, including rating schemas and text-generation assessment.

Compensation, schedule, and how it works

This is a contractor, part-time role with flexible hours. Expect to contribute 20+ hours per week and manage your own schedule within agreed experiment timelines.

• Pay: Hourly contract at $60–$90 USD per hour (posted rate: $90/hr; per-role range $60–$90/hr).
• Work type: Remote, US-only applicants.
• Data & tasks: Text-focused evaluation work including evaluation ratings and text-generation benchmark design.
• Employment: Contract / Part-time; OpenTrain AI is the hiring organization and project owner.