Senior ML Auditor for RLHF & Model Alignment (Remote)

prolificacademicltd; United States; Remote; USD 66,000 - 110,000; Full time

Benefits offered by this job

Remote work

Job summary

Prolific Academic Ltd is seeking Senior ML Engineers for a remote, paid expert network supporting large language model development and evaluation. You will audit AI explanations, review ML code, and provide structured feedback to improve alignment with human expectations.

Responsibilities include auditing training loops, evaluation scripts, and conducting benchmark comparisons to ensure technical fidelity and robustness in model behavior.

Qualifications

• BS, MS, or PhD in Computer Science, AI, Robotics, or a related quantitative field focused on machine learning.
• Track record of building, deploying, or fine-tuning ML models in production settings.
• Working command of neural network architectures (Transformers, CNNs, RNNs) and associated optimization techniques.
• Hands-on experience with prompt engineering, RLHF, or RAG pipelines.
• Ability to scrutinize complex model behavior, detect contamination, and assess mathematical underpinnings.
• Strong editorial eye for hallucinations, bias, and logical inconsistencies.

Responsibilities

• Critically review AI explanations of architectures, loss functions, and backpropagation for technical fidelity.
• Audit ML codebases and notebooks, including training loops and evaluation scripts, for correctness and performance.
• Deliver high-quality human judgments that drive RLHF alignment with respect to safety, helpfulness, and intent.
• Trace and critique chain-of-thought reasoning to surface breakdowns in model logic.
• Conduct systematic benchmark comparisons across model outputs using technical rubrics and metrics.

Skills

Transformers

Neural networks

RLHF

Prompt engineering

Code auditing

Education

BS/MS/PhD in CS/AI/ML

Tools

PyTorch

TensorFlow/Keras

Hugging Face Transformers

AWS SageMaker

Job description

Prolific Academic Ltd is seeking Senior ML Engineers for a remote, paid expert network supporting large language model development and evaluation. You will audit AI explanations, review ML code, and provide structured feedback to improve alignment with human expectations.

Responsibilities include auditing training loops, evaluation scripts, and conducting benchmark comparisons to ensure technical fidelity and robustness in model behavior.