Senior AI Evaluation & Benchmarking Engineer

OpenTeams; Tacoma (WA); Ibrido; USD 145.000 - 250.000; Tempo pieno

Descrizione del lavoro

OpenTeams is seeking a Senior AI Engineer - Evaluation & Benchmarking to build and operate the benchmarking capability at the core of a government AI platform. This role focuses on understanding model failures and surfacing critical risk areas to improve reliability.

You will design evaluation harnesses, combine human and AI judgments, and document limitations. You will report to senior stakeholders and help determine which capabilities are ready to field, working hands-on with an open-source

Competenze

• Proven experience building and deploying software platforms with complex integration surface, preferably ML or agentic workloads.
• Deep experience using python, including API frameworks and agentic harnesses.
• High level knowledge of computer vision, language modeling, and/or task-specific agent evaluation.
• Hands on experience with PyTorch and Hugging Face ecosystem.
• Strong communication skills and cross-functional collaboration with stakeholders.
• Demonstrated leadership in architectural decisions and standards.

Mansioni

• Design and implement a platform for test & evaluation of AI models and Agentic systems.
• Develop evaluation methodologies combining human and AI expert judging and multiple modalities.
• Design and enhance user interfaces for generic workflow expression by customers.
• Collaborate with subject matter experts and engineers to cover current and future requirements.
• Architect and implement robust experiment provenance and result tracking.
• Ensure performance across integrated tools and scalable systems.
• Provide mentorship, code reviews, design guidance, and documentation for the team.

Conoscenze

Python

API frameworks

Agentic harnesses

PyTorch

Hugging Face

ML platforms

Cross-functional collaboration

System design

Leadership

Communication

Strumenti

PyTorch

Hugging Face

Descrizione del lavoro

OpenTeams is seeking a Senior AI Engineer - Evaluation & Benchmarking to build and operate the benchmarking capability at the core of a government AI platform. This role focuses on understanding model failures and surfacing critical risk areas to improve reliability.

You will design evaluation harnesses, combine human and AI judgments, and document limitations. You will report to senior stakeholders and help determine which capabilities are ready to field, working hands-on with an open-source