Remote Senior AI/ML Benchmarking & Evaluation Engineer

OpenTeams; Northern (KY); Hybrid; USD 145,000 - 250,000; Full time

Job summary

OpenTeams seeks a Senior AI/ML Test and Evaluation Engineer to build and operate benchmarking and evaluation capabilities at the core of our AI platform. You will design automated metrics and human judgments to compare models and agentic workflows, surface failure modes, and ensure evaluations stay valid as models evolve.

This hands-on role uses an open‑source toolchain, reports to senior stakeholders, and helps decide which capabilities are ready to field, emphasizing robust, transparently

Qualifications

• Experience designing benchmarks for ML/agentic systems.
• Proficiency with Python for API frameworks and evaluation harnesses.
• Familiarity with ML/vision/language models and evaluation.

Responsibilities

• Design and implement a platform for test & evaluation of AI models and agentic systems.
• Develop evaluation methodologies combining human and AI expert judging and multiple modalities.
• Design and enhance user interfaces for workflow expression by customers.
• Collaborate with SMEs, model/agent developers, and evaluation designers.
• Architect and implement robust experiment provenance and result tracking.
• Ensure performance across integrated tools and scaling systems.
• Provide mentorship, code reviews, and documentation for knowledge transfer.

Skills

ML platforms

Python API

Computer vision / language modeling

PyTorch / HuggingFace

Cross-functional collaboration

Technical leadership

Tools

PyTorch

Hugging Face

Job description

OpenTeams seeks a Senior AI/ML Test and Evaluation Engineer to build and operate benchmarking and evaluation capabilities at the core of our AI platform. You will design automated metrics and human judgments to compare models and agentic workflows, surface failure modes, and ensure evaluations stay valid as models evolve.

This hands-on role uses an open‑source toolchain, reports to senior stakeholders, and helps decide which capabilities are ready to field, emphasizing robust, transparently