Remote AI Eval Engineer: Ground Truth & Datasets

Entrada Ventures

United States

Hybrid

USD 120,000 - 210,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Medical, dental & vision insurance
Equity and meaningful early-stage"
Unlimited PTO
Hybrid/Remote stipend
In-office perks and snacks

Job summary

Prophetic Software is seeking an engineer to build and defend evaluation datasets for a scalable ML stack. You will own ground truth definitions, data sourcing, labeling guidelines, and the evaluation loop, ensuring trusted data drives product decisions.

You’ll work with product and engineering to decompose systems into testable modules, define correctness, and measure performance. This role emphasizes hands-on data work and cross-team collaboration.

Qualifications

  • Experience building or evaluating ML/LLM systems in production.
  • You have built evaluation datasets and can discuss one in detail.
  • Working fluency in ML validation fundamentals: train/validation/test, precision/recall, calibration, inter-rater agreement.
  • Understand feature engineering to reason about classifier failures.
  • Strong Python and SQL; able to pull and reshape data yourself.
  • Hands-on with prompting, structured outputs, and tool-use harnesses for LLMs.
  • Judgment about when LLM-as-judge is reliable and how to prove it.
  • You can read a system design and translate it into a label schema.

Responsibilities

  • Decide what needs to be measured and how.
  • Define correctness with rubrics, label schemas, and tolerances.
  • Prioritize validation sets to unblock iterations.
  • Source data via production sampling, labeling programs, and synthetic ground truth.
  • Manage labeling vendors or internal experts and ensure label quality.
  • Calibrate automated graders against human gold sets.
  • Report metrics like precision, recall, F1, and calibration.
  • Collaborate with engineers to integrate datasets into pipelines.

Skills

ML/LLM systems in production
Evaluation datasets
Python
SQL
ML validation fundamentals
Labeling/vendor management
Data sampling
System design translation

Job description

Prophetic Software is seeking an engineer to build and defend evaluation datasets for a scalable ML stack. You will own ground truth definitions, data sourcing, labeling guidelines, and the evaluation loop, ensuring trusted data drives product decisions.

You’ll work with product and engineering to decompose systems into testable modules, define correctness, and measure performance. This role emphasizes hands-on data work and cross-team collaboration.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI Software Evaluator (Python/ML) — Contractor
Remote AI Software Evaluator (Python/ML) — Contractor

Turing • Los Angeles (CA)

Remote
USD 55,000 - 96,000
AI Evaluation & Measurement Engineer
AI Evaluation & Measurement Engineer

DEFCON AI, Inc. • United States

Remote
USD 120,000 - 170,000
Remote AI & ML Engineer: Training & Evaluation
Remote AI & ML Engineer: Training & Evaluation

Prolific • United States

Remote
USD 91,000 - 129,000
Remote AI Evaluation Engineer
Remote AI Evaluation Engineer

Planet Pharma • San Francisco (CA)

Remote
USD 120,000 - 180,000
Remote Research Engineer — AI Evaluation & RL Data Pipelines
Remote Research Engineer — AI Evaluation & RL Data Pipelines

Prime Intellect AI • San Francisco (CA)

Hybrid
USD 150,000 - 350,000
Salary + equity
Remote or SF Office
Visa sponsorship
+2
Remote AI QA Architect & Evaluation Lead
Remote AI QA Architect & Evaluation Lead

Meridial • United States

Remote
USD 8,300 - 90,000
Platform Engineer - RL & Evaluation Data Systems
Platform Engineer - RL & Evaluation Data Systems

anyone-ai • United States

Remote
USD 150,000 - 230,000
Remote AI Software Engineer - Python & ML Infra
Remote AI Software Engineer - Python & ML Infra

Turing • San Francisco (CA)

Remote
USD 83,000 - 165,000
Remote AI Software Evaluator – Full-Stack Engineer (Python)
Remote AI Software Evaluator – Full-Stack Engineer (Python)

Turing • Chicago (IL)

Remote
USD 83,000 - 165,000
Remote AI Evaluation Architect | Frontier ML Systems
Remote AI Evaluation Architect | Frontier ML Systems

Crossing Hurdles • United States

Remote
USD 240,000 - 350,000