Remote AI Engineer: LLM Evaluation & RLHF Audits

prolificacademicltd; United States; Remote; USD 66,000 - 110,000; Full time

Job summary

prolificacademicltd is seeking seasoned machine learning engineers to audit AI-generated content, code, and reasoning in short, remote sessions. The role involves evaluating explanations of architectures and training processes, and delivering high-impact feedback for RLHF pipelines.

Candidates should have production-level ML experience, a deep understanding of transformers/CNNs/RNNs, and practical exposure to prompt engineering and retrieval-augmented generation.

Qualifications

• Advanced degree with ML focus (BS/MS/PhD).
• Production-level experience with ML models: training, deployment or fine-tuning.
• Strong grasp of neural architectures (Transformers, CNNs, RNNs) and modern optimization methods.
• Familiarity with prompt engineering, RLHF or retrieval-augmented generation workflows.
• Meticulous attention to detail to catch hallucinations, bias, and logical flaws in AI output.

Responsibilities

• Evaluate accuracy of AI explanations on architectures, losses, and backpropagation.
• Audit ML codebases, training loops, and notebooks for correctness and performance.
• Provide high-quality human feedback for RLHF pipelines.
• Stress-test model reasoning on complex prompts and document failures.
• Compare model outputs using defined taxonomies and benchmarks.

Skills

ML model development

Prompt engineering

RLHF

Attention to detail

Audit ML outputs

Education

BS/MS/PhD in CS/AI/Robotics or related

Tools

PyTorch

TensorFlow

Keras

HuggingFace Transformers

AWS SageMaker

Vertex AI

Vector stores (Pinecone/Milvus/Weaviate)

Job description

prolificacademicltd is seeking seasoned machine learning engineers to audit AI-generated content, code, and reasoning in short, remote sessions. The role involves evaluating explanations of architectures and training processes, and delivering high-impact feedback for RLHF pipelines.

Candidates should have production-level ML experience, a deep understanding of transformers/CNNs/RNNs, and practical exposure to prompt engineering and retrieval-augmented generation.