AI Evaluation & Measurement Engineer

DEFCON AI, Inc.; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

DEFCON AI, Inc. is seeking an experienced professional to own and evolve the evaluation framework for AI-assisted systems. You will design robust audit and sampling methodologies and build labeled ground truth datasets to ensure reliable scoring across model releases.

You will measure performance, conduct drift and override reviews, and compile evidence packages to inform deployment decisions, collaborating with data scientists and engineers while remaining independent from build roles.

Responsibilities

• Own the evaluation framework for AI-assisted systems, building labeled ground truth datasets and designing statistically sound audit and sampling methodologies.
• Measure model performance across releases, run drift and override reviews, and produce evidence packages supporting deployment decisions.
• Work closely with data scientists and engineers to validate scoring and generative AI capabilities while maintaining independence from build roles.

Job description

DEFCON AI, Inc. is seeking an experienced professional to own and evolve the evaluation framework for AI-assisted systems. You will design robust audit and sampling methodologies and build labeled ground truth datasets to ensure reliable scoring across model releases.

You will measure performance, conduct drift and override reviews, and compile evidence packages to inform deployment decisions, collaborating with data scientists and engineers while remaining independent from build roles.