Remote AI Evaluation Data Engineer — Ground Truth

Propheticsoftware; Northern (KY); Hybrid; USD 120,000 - 190,000; Full time

Job summary

Prophetic Software is seeking an engineer to build and defend evaluation datasets for its AI-native platform. You will define correctness, design label schemas, and manage data quality across modules in production, collaborating with product and engineering teams.

You will be responsible for sourcing trustworthy ground truth through production sampling, labeling programs, and oracle-generated data, ensuring robust evaluation and measurable improvements.

Qualifications

• Experience building or evaluating ML or LLM-powered systems in production, where output quality was the focus.
• You have built evaluation or validation datasets and can discuss one in detail: definition of correctness, labeling process, and QA.
• Strong Python and SQL skills; able to pull and reshape data independently.
• Hands-on with prompting, structured outputs, and evaluation methods for LLM-based systems.

Responsibilities

• Decide what to measure and how, decomposing systems into evaluable modules with input → output contracts.
• Define correctness rubrics and edge-case policies that matter to the business.
• Source data using production sampling, labeling programs, and oracle-generated ground truth to build reliable eval sets.
• Calibrate automated graders against human gold sets and report metrics like precision/recall/F1.

Skills

Python

SQL

ML evaluation

Data analysis

Experiment design

Attention to detail

Tools

ML tooling

Job description

Prophetic Software is seeking an engineer to build and defend evaluation datasets for its AI-native platform. You will define correctness, design label schemas, and manage data quality across modules in production, collaborating with product and engineering teams.

You will be responsible for sourcing trustworthy ground truth through production sampling, labeling programs, and oracle-generated data, ensuring robust evaluation and measurable improvements.