Senior AI Model Auditor & Evaluator (Remote)

prolificacademicltd; United States; Remote; USD 91,000 - 129,000; Full time

Benefits offered by this job

Fully remote

Flexible scheduling

Project-based engagements

Job summary

Prolific Academic Ltd is seeking senior machine learning engineers to audit and evaluate cutting-edge large language models. You will work on remote, intermittent tasks that demand deep ML expertise and careful judgment about model behavior and outputs.

You will review architectures, validate training scripts, and provide structured feedback to improve RLHF alignment. Strong familiarity with Transformers and modern ML tooling is essential for this role.

Qualifications

• Degree in CS/AI/Robotics or related quantitative field with ML emphasis.
• Hands-on production experience building, deploying, or fine-tuning ML models.
• Strong grasp of neural networks including Transformers, CNNs, RNNs and modern optimizers.
• Experience with prompt engineering, RLHF, or retrieval-augmented generation.
• Proven ability to audit model behavior and detect data contamination or biased reasoning.
• Based in the Colorado Springs area.

Responsibilities

• Review AI-generated descriptions of architectures, loss functions, and backpropagation for correctness.
• Validate ML code, training scripts, and notebooks for efficiency and accuracy.
• Provide structured feedback to strengthen RLHF alignment processes.
• Probe model reasoning on chain-of-thought prompts and surface weaknesses.
• Benchmark outputs against technical taxonomies and quantitative criteria.

Skills

ML deployment

Transformer networks

Prompt engineering

RLHF

Model auditing

Bias/robustness

Education

Bachelor's/Master's/PhD in CS/AI/Robotics or related ML field

Tools

PyTorch

TensorFlow

Keras

NumPy

Pandas

Scikit-learn

Hugging Face Transformers

AWS SageMaker

Vertex AI

Weights & Biases

LangChain

Pinecone

Milvus

Weaviate

Job description

Prolific Academic Ltd is seeking senior machine learning engineers to audit and evaluate cutting-edge large language models. You will work on remote, intermittent tasks that demand deep ML expertise and careful judgment about model behavior and outputs.

You will review architectures, validate training scripts, and provide structured feedback to improve RLHF alignment. Strong familiarity with Transformers and modern ML tooling is essential for this role.