Senior AI/ML Benchmarking & Evaluation Engineer

openteams; Washington; Hybrid; USD 145,000 - 250,000; Full time

Job summary

OpenTeams is seeking a Senior AI/ML Test and Evaluation Engineer in a hybrid role based in Washington, DC or CO locations. You will build and operate evaluation harnesses and automated metrics alongside expert judgment to stress-test models and agentic workflows.

You will document limitations, surface critical failure modes, and produce defensible reports for senior stakeholders. This hands-on engineering role uses open-source toolchains and may require travel up to 15% to government facilities

Qualifications

• US citizenship and eligibility to obtain and maintain a US security clearance
• 6+ years of software engineering or ML engineering experience, including 3+ years evaluating, benchmarking, or deploying ML models in prod or applied research
• Strong Python proficiency in ML/data science context
• Hands-on experience with ML frameworks and tooling such as PyTorch and Hugging Face ecosystem
• Experience developing or using model evaluation harnesses, benchmark suites, or test/evaluation frameworks
• Experience designing evaluation metrics with statistical rigor
• Experience building repeatable, auditable evaluation pipelines with data provenance
• Experience evaluating LLMs or agentic workflows using task-, metric-, or judgment-based scoring
• Strong written communication for explaining evaluation methodologies and results
• Ability to translate mission needs into practical evaluation approaches

Responsibilities

• Design, implement, and operate benchmark execution and evaluation harnesses for AI models and agentic workflows
• Develop evaluation methodologies combining automated metrics with structured human expert judgment
• Curate and recommend candidate benchmarks based on mission needs and document provenance of ground-truth data
• Produce defensible evaluation reports comparing candidate capabilities with current mission workflows and document limitations
• Define and contribute to standards for benchmark expression, ingestion, and reporting
• Support partner organizations and vendors as they integrate with shared evaluation standards
• Build lightweight expert-scoring workflows and measure inter-reviewer agreement
• Participate in feedback sessions with mission end users and incorporate findings
• Develop reference notebooks and example workflows for data-science-capable analysts
• Document technical approaches, evaluation results, and key decisions for government stakeholders and teams

Skills

Python

PyTorch

Hugging Face

ML Evaluation

Benchmarking

Software Engineering

Security Clearance

US Citizenship

Education

Bachelor's degree in computer science, mathematics, engineering, or related field

Tools

Evaluation harnesses

Benchmark suites

Job description

OpenTeams is seeking a Senior AI/ML Test and Evaluation Engineer in a hybrid role based in Washington, DC or CO locations. You will build and operate evaluation harnesses and automated metrics alongside expert judgment to stress-test models and agentic workflows.

You will document limitations, surface critical failure modes, and produce defensible reports for senior stakeholders. This hands-on engineering role uses open-source toolchains and may require travel up to 15% to government facilities