AI Model Evaluation Data Scientist

OpenTrain AI, Inc.; United States; Remote; USD 34,000 - 55,000; Part time

Job summary

OpenTrain AI, Inc. is seeking an entry-level contractor to join our AI training program. You will design, implement, and evaluate Python-based data analysis workflows, and create datasets and clear explanations of results.

You will collaborate with researchers and annotators on supervised fine-tuning, RLHF, and reward-model refinement. Remote work, global applicants, with 20–40 hours weekly and overlap with Pacific Time.

Qualifications

• Bachelor's or Master's degree in engineering or computer science (or equivalent).
• Entry-level experience in Python and data analysis.
• Strong problem-solving ability and sound business judgment.

Responsibilities

• Design, develop, and maintain Python code for training and optimizing AI models.
• Benchmark model performance and analyze evaluation results.
• Evaluate and rank model responses to user queries using set criteria.
• Write clear explanations and rationales for evaluation decisions.
• Create and maintain datasets for supervised fine-tuning.
• Create and refine responses for clarity, relevance, and technical accuracy.
• Review code and documentation and provide constructive feedback.
• Use public datasets from sources such as Kaggle, the United Nations, and the US government.

Skills

Python

Data analysis

Problem-solving

Business judgment

Communication

Education

Bachelor's or Master’s in Engineering/CS

Tools

Jupyter notebooks

Job description

The work

You will use Python and data analysis to evaluate AI models and identify ways to improve their performance. The work combines technical analysis, response review, dataset creation, and clear written reasoning.

You will work with researchers and annotators on supervised fine-tuning, reinforcement learning with human feedback, and reward-model refinement. You will also use public datasets to answer business questions and review code and documentation for issues.

• Design, develop, and maintain Python code for training and optimizing AI models.
• Benchmark model performance and analyze evaluation results.
• Evaluate and rank model responses to user queries using set criteria.
• Write clear explanations and rationales for evaluation decisions.
• Create and maintain datasets for supervised fine-tuning.
• Create and refine responses for clarity, relevance, and technical accuracy.
• Review code and documentation and provide constructive feedback.
• Use public datasets from sources such as Kaggle, the United Nations, and the US government.

What it pays and takes

The listing does not specify a pay rate. This is a remote contractor assignment with a one-month contract term.

• Hours: At least 20 hours per week, with options for 20, 30, or 40 hours per week.
• Schedule: Four hours of daily overlap with Pacific Time.
• Location: Fully remote and open worldwide.
• Language: Fluent conversational and written English.
• Education: A bachelor's or master's degree in engineering or computer science, or equivalent experience.
• Experience level: Entry level.
• Requirements: Proficiency in Python, strong data analysis skills, problem-solving ability, and sound business judgment.
• Communication: Explain technical reasoning clearly in Jupyter notebooks or comparable formats and communicate effectively with researchers and other stakeholders.

About AI training work

OpenTrain is the hiring and contracting organization for this role. AI training work uses human examples and feedback to improve how artificial intelligence systems respond, and people with data science and technical skills help assess model quality and guide those improvements.