Senior AI Model Auditor & RLHF Evaluator (Remote)

prolificacademicltd; United States; Remote; USD 91,000 - 129,000; Full time

Benefits offered by this job

Hourly pay up to $80

Fully remote with flexible hours

Brief onboarding

Job summary

prolificacademicltd is seeking experienced AI/ML engineers for a remote, flexible contributor-track role focused on training and evaluating large language models. You will review explanations, audit training code, and provide high-quality feedback used in RLHF to align models with human intent.

Ideal candidates have advanced degrees in CS/AI, hands-on experience with Transformer-based models, and proficiency in PyTorch or TensorFlow.

Qualifications

• Bachelor’s, Master’s or PhD in CS/AI/related quantitative field with ML emphasis.
• Professional experience building, deploying, or fine-tuning ML models in production.
• Deep familiarity with Transformer, CNN, RNN architectures and optimizations.
• Hands-on prompt engineering, RLHF pipelines, or retrieval-augmented generation (RAG).
• Sharp attention to detail to catch hallucinations, biases, or logical AI failures.

Responsibilities

• Review AI-generated explanations of model architectures, losses, and backpropagation for accuracy.
• Audit ML code and notebooks: training loops, preprocessing scripts, and evaluation routines.
• Provide high-quality human feedback used in RLHF to align models with human intent and safety.
• Analyze model behavior on chain-of-thought prompts and flag breakdowns in reasoning.
• Conduct comparative testing across model outputs using defined technical criteria.

Skills

ML research

prompt engineering

code auditing

RLHF evaluation

attention to detail

Education

BS/MS/PhD in CS/AI/related field

Tools

PyTorch

TensorFlow/Keras

AWS SageMaker

LangChain

Pinecone/Milvus/Weaviate

Job description

prolificacademicltd is seeking experienced AI/ML engineers for a remote, flexible contributor-track role focused on training and evaluating large language models. You will review explanations, audit training code, and provide high-quality feedback used in RLHF to align models with human intent.

Ideal candidates have advanced degrees in CS/AI, hands-on experience with Transformer-based models, and proficiency in PyTorch or TensorFlow.