Data Science QA Lead

OpenTrain AI; Northern (KY); Hybrid; USD 91,000 - 152,000; Part time

Benefits offered by this job

Remote hourly contractor role

Job summary

OpenTrain AI is seeking a Data Science QA Lead to review AI-generated data science content and trainer QA work. You will assess statistical accuracy, model selection, code correctness, reproducibility, metric interpretation, business context, instruction following, and rubric adherence.

This remote hourly contractor role is for US-based contributors working 20+ hours per week. The advertised rate is up to $110 per hour, with a focus on precise written feedback and improving QA processes.

Qualifications

• Degree in data science, statistics, computer science, machine learning, or related quantitative field.
• Strong English communication for clear technical feedback and team coordination.
• At least 3 years of experience in data science, analytics, ML, or related fields.
• Strong understanding of statistics, model evaluation, experimentation, and validation methods.
• Familiarity with Python, pandas, NumPy, scikit-learn, SQL, Jupyter, and related tools.
• Experience with AI training, data annotation, LLM evaluation, or rubric-based review is a plus.

Responsibilities

• Review AI-generated data science explanations, Python, R, and SQL snippets, modeling workflows, dashboards, and experiments.
• Check for data leakage, flawed assumptions, incorrect metrics, weak methodology, non-reproducible code, and misleading conclusions.
• Assess analytical work against rubrics covering statistical accuracy, model selection, code correctness, business context, and instruction following.
• Communicate guideline changes and workflow updates to trainers and QAs.
• Create and maintain style guides, trackers, FAQs, examples, calibration tasks, and onboarding materials.
• Support onboarding and training calls for contributors.
• Identify recurring issues and help improve QA processes.

Skills

Data science

Statistics

Machine learning

English communication

Rubric-based review

Education

Degree in data science or related quantitative field

Tools

Python

SQL

R

Pandas

NumPy

scikit-learn

Jupyter

Matplotlib

Spark

Git

MLflow

Job description

About OpenTrain

OpenTrain AI is the hiring and contracting organization for this role. OpenTrain is the #1 platform for finding and building careers in AI training and data labeling, helping contributors discover projects, build a professional profile, and apply to opportunities in minutes.

Creating an OpenTrain account is free, and this opportunity offers a way to contribute directly to the development and evaluation of modern AI systems.

About AI Training and Data Evaluation

AI training is the human side of building artificial intelligence. People review examples, evaluate model outputs, check technical accuracy, and provide feedback that helps AI systems become more useful and reliable.

In this role, your data science expertise will support evaluation of AI-generated explanations, analytical workflows, code, experiments, and conclusions. The work is remote and can be a flexible way to apply specialized skills to cutting-edge AI projects.

The Data Science QA Lead Role

OpenTrain AI is recruiting a Data Science QA Lead to review AI-generated data science content and trainer QA work. You will assess statistical accuracy, model selection, code correctness, reproducibility, metric interpretation, business context, instruction following, and rubric adherence.

You will also provide precise written feedback, identify recurring quality issues, help update trainers and QAs, support contributor onboarding, maintain quality documentation, and improve QA processes. This is a remote hourly contractor role for US-based contributors working 20 or more hours per week.

• Advertised rate of up to $110 per hour
• Part-time contractor engagement
• Remote work available in the United States
• English-language role requiring strong written communication

What You'll Do

You will evaluate both the technical substance and communication quality of data science work. Your reviews will help ensure that AI-generated content is accurate, reproducible, methodologically sound, and aligned with project requirements.

• Review AI-generated data science explanations, Python, R, and SQL snippets, modeling workflows, dashboards, experiment designs, and step-by-step reasoning.
• Check for data leakage, flawed assumptions, incorrect metrics, weak methodology, non-reproducible code, and misleading conclusions.
• Assess analytical work against rubrics covering statistical accuracy, model selection, code correctness, business context, instruction following, and rubric adherence.
• Communicate guideline changes and workflow updates to trainers and QAs.
• Create and maintain style guides, trackers, FAQs, examples, honeypots, calibration tasks, and onboarding materials.
• Support onboarding and training calls for contributors.
• Identify recurring issues and help improve QA processes.

Required Qualifications and Skills

The role requires a strong quantitative background and the ability to review analytical work against detailed rubrics. Candidates should be comfortable explaining technical findings clearly in written English and coordinating with distributed contributors.

• Degree in data science, statistics, computer science, machine learning, mathematics, economics, engineering, or a related quantitative field.
• Strong English communication skills for clear technical feedback and team coordination.
• At least 3 years of experience in data science, analytics, machine learning, statistical modeling, experimentation, data engineering, technical review, or data science education.
• Strong understanding of statistics, model evaluation, experimentation, regression, classification, clustering, and validation methods.
• Familiarity with Python, pandas, NumPy, scikit-learn, SQL, Jupyter, matplotlib, R, Spark, Git, MLflow, notebooks, dashboards, and cloud or data platforms.
• Experience with AI training, data annotation, LLM evaluation, data science QA, or rubric-based technical review is a strong plus.

Helpful Background

Experience supporting distributed teams and maintaining clear quality resources will help you succeed. The work involves coordinating updates, organizing documentation, and keeping review standards consistent across projects.

• Experience leading or supporting remote teams of trainers, annotators, analysts, data scientists, engineers, educators, or QAs.
• Comfort using Discord, Google Sheets, Google Docs, trackers, dashboards, GitHub, and project management systems.
• Strong organizational skills and the ability to maintain documentation and quality resources.
• Availability for 20 or more hours per week.