Senior AI/ML Evaluator for LLMs - Remote & Flexible

prolificacademicltd; United States; Remote; USD 83,000 - 110,000; Full time

Benefits offered by this job

Fully remote

Flexible hours

Per-task payments

Job summary

Prolific Academic Ltd invites Senior AI and Machine Learning Engineers to join an expert contributor network that trains and evaluates large language models for AI researchers and developers. Contributors are paid per task, work remotely on a flexible schedule, and focus on short, highly technical assignments that leverage deep ML expertise rather than full-time employment.

The role emphasizes careful analysis of model reasoning, RLHF workflows, and benchmarking across model outputs to improve

Qualifications

• BS, MS, or PhD in Computer Science, Artificial Intelligence, Robotics, or a related quantitative field with a focus on Machine Learning.
• Hands-on experience building, deploying, or fine-tuning machine learning models in a production environment.
• Professional-level understanding of neural network architectures such as Transformers, CNNs, and RNNs, plus modern optimisation techniques.
• Practical experience with prompt engineering, RLHF, or retrieval-augmented generation workflows.
• Ability to audit complex model logic, detect training data contamination, and evaluate mathematical proofs behind ML algorithms.
• High attention to detail when spotting hallucinations, biased outputs, or logical failures in AI-generated technical content.

Responsibilities

• Review AI-generated explanations of model architectures, loss functions, and backpropagation for technical accuracy.
• Audit machine learning code and notebooks, including training loops, preprocessing scripts, and evaluation pipelines, for correctness and efficiency.
• Provide structured human feedback used to refine RLHF frameworks and align model behaviour with human intent, safety, and helpfulness.
• Analyse how models navigate complex chain-of-thought prompts and pinpoint where reasoning breaks down.
• Run comparative benchmark tests across model outputs using defined taxonomies and performance metrics.

Skills

Transformers

Prompt engineering

RLHF

Python (NumPy, Pandas)

Auditing ML models

Education

BS/MS/PhD in CS/AI/Robotics or related ML field

Tools

PyTorch

TensorFlow/Keras

Hugging Face Transformers

AWS SageMaker

Weights & Biases

LangChain

Pinecone/Milvus/Weaviate

Job description

Prolific Academic Ltd invites Senior AI and Machine Learning Engineers to join an expert contributor network that trains and evaluates large language models for AI researchers and developers. Contributors are paid per task, work remotely on a flexible schedule, and focus on short, highly technical assignments that leverage deep ML expertise rather than full-time employment.

The role emphasizes careful analysis of model reasoning, RLHF workflows, and benchmarking across model outputs to improve