Remote Model Evaluation & Validation Engineer

DEFCON AI, Inc.; United States; Remote; USD 150,000 - 190,000; Full time

Benefits offered by this job

Fully remote role

Equity package

Health insurance for you and family

Unlimited PTO

Flexible work environment

14 weeks of fully-paid parental leave

Job summary

DEFCON AI, Inc. is seeking an experienced Model Test & Measurement Engineer for an AI-enabled platform in a secure government cloud environment.

You will own the evaluation framework, build labeled ground truth datasets, and validate scoring and generative AI capabilities to provide trusted, explainable evidence for deployment decisions. You will work closely with data scientists and AI engineers while maintaining independence from model building and threshold approvals.

Qualifications

• 5+ years with model validation as a named responsibility, not a side task.
• Direct experience with ground-truth construction and sampling design.
• Strong Python and SQL; comfort working independently from the teams whose models you evaluate.
• Elevated personnel security requirements apply to portions of this work and are discussed during screening.

Responsibilities

• Construct labeled ground truth for model evaluation.
• Design and run sampled audits.
• Gate releases against model version; maintain version inventory, evaluation records, and rollback triggers.
• Run drift and override review.
• Produce human-oversight and fairness / disparate-effect evidence.
• Maintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence.
• Measure workflow improvement: review time, throughput, backlog movement, and override and rework rates.
• Design evaluation-phase QC: sample selection that does not mix populations, the unit of review, and fair comparison when methods or searched sources differ.
• Define the measurement events other teams must emit, and confirm the review workspace emits workflow events from first use.

Skills

5+ years model validation

Ground-truth construction

Python

SQL

Job description

DEFCON AI, Inc. is seeking an experienced Model Test & Measurement Engineer for an AI-enabled platform in a secure government cloud environment.

You will own the evaluation framework, build labeled ground truth datasets, and validate scoring and generative AI capabilities to provide trusted, explainable evidence for deployment decisions. You will work closely with data scientists and AI engineers while maintaining independence from model building and threshold approvals.