Senior AI Engineer - Evaluation & Benchmarking

OpenTeams; Tacoma (WA); Híbrido; USD 145.000 - 250.000; Jornada completa

Descripción de la vacante

OpenTeams is seeking a Senior AI Engineer - Evaluation & Benchmarking to build and operate the benchmarking capability at the core of a government AI platform. This role focuses on understanding model failures and surfacing critical risk areas to improve reliability.

You will design evaluation harnesses, combine human and AI judgments, and document limitations. You will report to senior stakeholders and help determine which capabilities are ready to field, working hands-on with an open-source

Formación

• Proven experience building and deploying software platforms with complex integration surface, preferably ML or agentic workloads.
• Deep experience using python, including API frameworks and agentic harnesses.
• High level knowledge of computer vision, language modeling, and/or task-specific agent evaluation.
• Hands on experience with PyTorch and Hugging Face ecosystem.
• Strong communication skills and cross-functional collaboration with stakeholders.
• Demonstrated leadership in architectural decisions and standards.

Responsabilidades

• Design and implement a platform for test & evaluation of AI models and Agentic systems.
• Develop evaluation methodologies combining human and AI expert judging and multiple modalities.
• Design and enhance user interfaces for generic workflow expression by customers.
• Collaborate with subject matter experts and engineers to cover current and future requirements.
• Architect and implement robust experiment provenance and result tracking.
• Ensure performance across integrated tools and scalable systems.
• Provide mentorship, code reviews, design guidance, and documentation for the team.

Conocimientos

Python

API frameworks

Agentic harnesses

PyTorch

Hugging Face

ML platforms

Cross-functional collaboration

System design

Leadership

Communication

Herramientas

PyTorch

Hugging Face

Descripción del empleo

Location: U.S Hybrid - Washington, DC, Denver, CO or Colorado Springs, CO.

Clearance Requirement: An active TS/SCI clearance is preferred. Candidates may also be considered if they previously held a TS/SCI with CI polygraph or currently hold an active TS or Secret clearance. Candidates that are not eligible or willing to obtain a security clearance will not be considered.

Salary Range: $145,000-$250,000 USD, dependent on experience level and location

Travel Requirement: Up to 15% travel is required.

About the Role

We're looking for a Senior AI Engineer - Evaluation & Benchmarking to build and operate the benchmarking and evaluation capability at the core of a government AI platform. This is a role for someone who is more interested in what a model gets wrong than in what it gets right.

You build the evaluation harnesses - automated metrics paired with structured human expert judgment, applied to candidate models and to the agentic workflows built on top of them. You develop repeatable methodologies for comparing performance against current operational baselines, which means the comparison holds up when someone runs it again in six months with a different model. And you document the limitations and surface the failure modes that matter, including the ones nobody asked about.

Your reports go to senior stakeholders and inform decisions about which capabilities are ready to field. That's the weight of the job: a benchmark that looks good and hides a failure mode is worse than no benchmark at all, and you're the check against that.

This is hands-on engineering on an open-source toolchain.

Key Responsibilities

• Design and implement a platform for test & evaluation of AI models and Agentic systems
• Develop evaluation methodologies combining human and AI expert judging and multiple input and output modalities
• Design and enhance user interfaces such that platform customers can express workflows in an intuitive and generic manner
• Collaborate closely with subject matter experts, Model/Agent developers, and evaluation designers to ensure coverage over existing and to be discovered requirements.
• Architect and implement robust experiment provenance and result tracking systems.
• Ensure performance across integrated tools and scaling systems.
• Provide technical mentorship, code review, design guidance, and clear documentation to support team knowledge transfer and user onboarding.

Required Skills & Experience

• Proven experience building and deploying software platforms with complex integration surface, preferably handling Machine Learning or agentic workloads.
• Deep experience using python, including API frameworks and agentic harnesses.
• High level knowledge of Computer vision, language modeling, and/or task specific agent evaluation.
• Hands on experience with common frameworks and tooling, eg PyTorch and Hugging Face ecosystem.
• Strong communication skills and experience collaborating effectively with cross functional teams including external stakeholders.
• Demonstrated experience or potential for leadership, especially in making architectural decisions, coordinating the efforts of other engineers, and developing standards and best practices.

Nice to Have

• Currently hold or eligible to hold U.S. Security Clearance (Secret or higher)
• Prior experience designing test and evaluation systems for AI
• Experience deploying complex software systems in IL 4 or higher environments.
• Experience participating in security assessment and authorization, hardening systems, and remediating vulnerabilities.
• Previous experience contributing to open source projects and participating in open source communities.
• Exposure to the intelligence community or department of defense subject matter experts.