Evals Engineer

PingAura AI; United States; À distance; USD 120 000 - 170 000; Plein temps

Résumé du poste

PingAura AI is seeking a role focused on building and maintaining the evaluation process for an AI assistant. The role centers on measuring performance, detecting regressions early, and ensuring evaluation results influence release decisions.

You will own the evaluation suite, implement checks to reveal quality changes over time, and ensure regressions are caught before customers are affected. This position emphasizes AI evaluation and release gating within a fast-paced product environment.

Qualifications

• Experience with AI evaluation concepts and metrics.
• Ability to define and maintain evaluation gates for releases.
• Proficiency in monitoring model performance over time.

Responsabilités

• Own the evaluation suite used to measure assistant performance.
• Create or maintain checks that reveal quality changes over time.
• Identify regressions before they reach customers.
• Make evaluation results a required gate in the release process.

Connaissances

AI evaluation

Performance measurement

Regression detection

Release gating

Description du poste

Role overview

Build and maintain the evaluation process for an AI assistant. The role focuses on measuring how well it performs, detecting regressions early, and making evaluation results part of release decisions.

Responsibilities

• Own the evaluation suite used to measure assistant performance.
• Create or maintain checks that reveal quality changes over time.
• Identify regressions before they reach customers.
• Make evaluation results a required gate in the release process.

Requirements

• The source emphasizes AI evaluation, performance measurement, regression detection, and release gating. It does not specify required experience, education, or particular technologies.