Remote AI Eval Engineer: Ground Truth & Datasets

Entrada Ventures; United States; Hybrid; USD 120,000 - 210,000; Full time

Benefits offered by this job

Medical, dental & vision insurance

Equity and meaningful early-stage"

Unlimited PTO

Hybrid/Remote stipend

In-office perks and snacks

Job summary

Prophetic Software is seeking an engineer to build and defend evaluation datasets for a scalable ML stack. You will own ground truth definitions, data sourcing, labeling guidelines, and the evaluation loop, ensuring trusted data drives product decisions.

You’ll work with product and engineering to decompose systems into testable modules, define correctness, and measure performance. This role emphasizes hands-on data work and cross-team collaboration.

Qualifications

• Experience building or evaluating ML/LLM systems in production.
• You have built evaluation datasets and can discuss one in detail.
• Working fluency in ML validation fundamentals: train/validation/test, precision/recall, calibration, inter-rater agreement.
• Understand feature engineering to reason about classifier failures.
• Strong Python and SQL; able to pull and reshape data yourself.
• Hands-on with prompting, structured outputs, and tool-use harnesses for LLMs.
• Judgment about when LLM-as-judge is reliable and how to prove it.
• You can read a system design and translate it into a label schema.

Responsibilities

• Decide what needs to be measured and how.
• Define correctness with rubrics, label schemas, and tolerances.
• Prioritize validation sets to unblock iterations.
• Source data via production sampling, labeling programs, and synthetic ground truth.
• Manage labeling vendors or internal experts and ensure label quality.
• Calibrate automated graders against human gold sets.
• Report metrics like precision, recall, F1, and calibration.
• Collaborate with engineers to integrate datasets into pipelines.

Skills

ML/LLM systems in production

Evaluation datasets

Python

SQL

ML validation fundamentals

Labeling/vendor management

Data sampling

System design translation

Job description

Prophetic Software is seeking an engineer to build and defend evaluation datasets for a scalable ML stack. You will own ground truth definitions, data sourcing, labeling guidelines, and the evaluation loop, ensuring trusted data drives product decisions.

You’ll work with product and engineering to decompose systems into testable modules, define correctness, and measure performance. This role emphasizes hands-on data work and cross-team collaboration.