Senior AI/ML Auditor for Frontier LLMs (Remote)

prolificacademicltd; United States; Remote; USD 66,000 - 110,000; Full time

Benefits offered by this job

Remote work

Flexible hours

Technical assessment

Job summary

prolificacademic ltd seeks a senior AI/ML engineer to evaluate frontier LLMs, audit training code, and critique model outputs. You will focus on deep technical judgment and error tracing rather than building models, working remotely with flexible hours in the US.

You will analyze prompts, assist in RLHF workflows, and benchmark reasoning quality using defined metrics and taxonomies. A strong ML background is required.

Qualifications

• BS, MS, or PhD in Computer Science, AI, Robotics, or a quantitative field with ML concentration.

Responsibilities

• Scrutinize AI explanations of architectures, loss functions, and backpropagation for accuracy.
• Audit ML code including training loops, data preprocessing, and evaluation notebooks for issues.
• Provide high-quality feedback for RLHF targeting alignment, safety, and helpfulness.
• Trace AI reasoning through chain-of-thought prompts to locate logical failures.
• Conduct benchmarks across model outputs using defined taxonomies and metrics.

Skills

Machine learning

Production ML

Code auditing

Analytical reasoning

Prompt engineering

RLHF

Retrieval augmented generation

Python

Neural networks

Education

Bachelor's/Master's/PhD in CS/AI/Robotics

Tools

PyTorch

TensorFlow/Keras

NumPy

Pandas

Scikit-learn

Hugging Face Transformers

Pinecone

Milvus

Weaviate

AWS SageMaker

Job description

prolificacademic ltd seeks a senior AI/ML engineer to evaluate frontier LLMs, audit training code, and critique model outputs. You will focus on deep technical judgment and error tracing rather than building models, working remotely with flexible hours in the US.

You will analyze prompts, assist in RLHF workflows, and benchmark reasoning quality using defined metrics and taxonomies. A strong ML background is required.