Remote AI Model Validation & Evaluation Engineer

DEFCON AI, Inc.; McLean (VA); Remote; USD 150,000 - 190,000; Full time

Benefits offered by this job

Fully remote

Equity package

Health insurance

Unlimited PTO

Flexible work schedule

14 weeks parental leave

Job summary

DEFCON AI, Inc. is seeking a Model Test & Measurement Engineer to lead evaluation frameworks for an AI-assisted platform in a secure government cloud. You will build ground truth datasets, design audits, and validate scoring and generative AI capabilities.

You will work with data scientists and AI engineers, maintain independence, and influence release decisions. This is a fully remote role with occasional travel to headquarters or sites.

Qualifications

• 5+ years with model validation as a named responsibility.
• Direct experience with ground-truth construction and sampling design.
• Strong Python and SQL; comfort working independently from the teams whose models you evaluate.
• US Citizenship required.
• Active US Secret clearance.
• Elevated personnel security requirements apply to portions of this work and are discussed during screening.

Responsibilities

• Construct labeled ground truth for model evaluation.
• Design and run sampled audits.
• Gate releases against model version; maintain version inventory, evaluation records, and rollback triggers.
• Run drift and override review.
• Produce human-oversight and fairness / disparate-effect evidence.
• Maintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence.
• Measure workflow improvement: review time, throughput, backlog movement, and override and rework rates.
• Design evaluation-phase QC: sample selection that does not mix populations, the unit of review, and fair comparison when methods or searched sources differ.
• Define the measurement events other teams must emit, and confirm the review workspace emits workflow events from first use.

Skills

Model validation

Ground-truth construction

Sampling design

Python

SQL

US Citizenship

Secret clearance

Security clearance process

Job description

DEFCON AI, Inc. is seeking a Model Test & Measurement Engineer to lead evaluation frameworks for an AI-assisted platform in a secure government cloud. You will build ground truth datasets, design audits, and validate scoring and generative AI capabilities.

You will work with data scientists and AI engineers, maintain independence, and influence release decisions. This is a fully remote role with occasional travel to headquarters or sites.