Remote AI Model Validation Engineer — Ground Truth & Metrics

Openkrill; United States; Remote; USD 150,000 - 190,000; Full time

Benefits offered by this job

100% remote work

Competitive salary and equity

Health insurance including family

Unlimited PTO

Job summary

DEFCON AI is seeking a Model Test & Measurement Engineer to own evaluation frameworks for AI systems in a secure government cloud environment. You will build labeled ground truth datasets, design audit methodologies, and validate both scoring and generative capabilities while remaining independent from model building and threshold decisions.

You will influence release decisions with objective evidence, collaborate with data scientists and AI engineers, and travel occasionally to DEFCON AI

Qualifications

• 5+ years with model validation as a named responsibility
• Direct experience with ground-truth construction and sampling design
• Strong Python and SQL; comfort working independently from the teams whose models you evaluate
• US Citizenship is required and Active US Secret clearance is required

Responsibilities

• Construct labeled ground truth for model evaluation
• Design and run sampled audits
• Gate releases against model version; maintain version inventory, evaluation records, and rollback triggers
• Run drift and override review
• Produce human-oversight and fairness / disparate-effect evidence
• Maintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence
• Measure workflow improvement: review time, throughput, backlog movement, and override and rework rates
• Design evaluation-phase QC: sample selection that does not mix populations, the unit of review, and fair comparison when methods or searched sources differ
• Define the measurement events other teams must emit, and confirm the review workspace emits workflow events from first use

Skills

Model validation

Ground-truth construction

Python

SQL

US citizenship

Secret clearance

Security compliance

Job description

DEFCON AI is seeking a Model Test & Measurement Engineer to own evaluation frameworks for AI systems in a secure government cloud environment. You will build labeled ground truth datasets, design audit methodologies, and validate both scoring and generative capabilities while remaining independent from model building and threshold decisions.

You will influence release decisions with objective evidence, collaborate with data scientists and AI engineers, and travel occasionally to DEFCON AI