Remote AI Model Auditor & RLHF Evaluator

prolificacademicltd; United States; Remote; USD 66,000 - 110,000; Full time

Benefits offered by this job

Up to $80/hr

Fully remote

Skills assessment gate

Job summary

Prolific Academic Ltd. seeks experienced AI/ML engineers to join a paid expert network evaluating and improving language model outputs.

This is a flexible, remote role based in the Mesa area, with hourly compensation for completed tasks and a focus on model alignment and reasoning quality. Contributors review model explanations, audit code and notebooks, and provide structured feedback to refine RLHF processes.

Qualifications

• BS, MS, or PhD in Computer Science, AI, Robotics, or related quantitative field with ML focus.
• Hands-on experience building, deploying, or fine-tuning ML models in production.
• Knowledge of transformers, CNNs, and RNNs with relevant optimization techniques.
• Experience with prompt engineering, RLHF, or RAG workflows.
• Ability to reason about ML proofs and audit complex model logic.
• Strong attention to detail for detecting hallucinations and biased outputs.

Responsibilities

• Review AI explanations of architectures, loss functions, and backpropagation for technical accuracy.
• Audit ML code/notebooks for correctness and efficiency.
• Provide structured human feedback to refine RLHF and improve alignment.
• Evaluate how a model reasons through chain-of-thought prompts and identify breakdowns.
• Run benchmarks across outputs against technical taxonomies and criteria.
• Identify hallucinations, bias, data contamination, and flawed reasoning.

Skills

Machine Learning in production

Transformers

RNNs/CNNs

RLHF

Prompt engineering

Code review

Education

Machine Learning degree

Tools

PyTorch

TensorFlow/Keras

Hugging Face Transformers

AWS SageMaker

LangChain

Pinecone / Milvus / Weaviate

Job description

Prolific Academic Ltd. seeks experienced AI/ML engineers to join a paid expert network evaluating and improving language model outputs.

This is a flexible, remote role based in the Mesa area, with hourly compensation for completed tasks and a focus on model alignment and reasoning quality. Contributors review model explanations, audit code and notebooks, and provide structured feedback to refine RLHF processes.