AI Evaluation Architect

Openkrill; United States; Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Medical, dental & vision insurance

Equity

Unlimited PTO

Hybrid/Remote stipend

In-office perks

Job summary

Prophetic seeks an engineer to own evaluation datasets and ground truth for its AI-native platform. You’ll define correctness, source data through diverse methods, and build robust evaluation pipelines to measure model and system performance.

You’ll collaborate with product, engineering, and ML teams to ensure data quality, reproducibility, and transparency in evaluation across the stack.

Qualifications

• Experience building or evaluating ML/LLM-powered systems in production.
• Built evaluation or validation datasets with defined correctness and labeled data.

Responsibilities

• Read pipelines and architecture; decompose into evaluable modules with input→output contracts.
• Define correctness rubrics, label schemas, edge-case policies, and tolerances.
• Prioritize modules to maximize iteration with minimal guidance from engineers.
• Source data via production sampling, human labeling, or synthetic ground truth.
• Calibrate automated graders against human gold sets; report metrics such as precision/recall.

Skills

Python

SQL

ML validation

LLM‑based systems

Tools

Eval tooling

Labeling platforms

Job description

Prophetic seeks an engineer to own evaluation datasets and ground truth for its AI-native platform. You’ll define correctness, source data through diverse methods, and build robust evaluation pipelines to measure model and system performance.

You’ll collaborate with product, engineering, and ML teams to ensure data quality, reproducibility, and transparency in evaluation across the stack.