AI Evaluation & Release Gate Engineer

PingAura AI; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

PingAura AI is seeking a role focused on building and maintaining the evaluation process for an AI assistant. The role centers on measuring performance, detecting regressions early, and ensuring evaluation results influence release decisions.

You will own the evaluation suite, implement checks to reveal quality changes over time, and ensure regressions are caught before customers are affected. This position emphasizes AI evaluation and release gating within a fast-paced product environment.

Qualifications

• Experience with AI evaluation concepts and metrics.
• Ability to define and maintain evaluation gates for releases.
• Proficiency in monitoring model performance over time.

Responsibilities

• Own the evaluation suite used to measure assistant performance.
• Create or maintain checks that reveal quality changes over time.
• Identify regressions before they reach customers.
• Make evaluation results a required gate in the release process.

Skills

AI evaluation

Performance measurement

Regression detection

Release gating

Job description

PingAura AI is seeking a role focused on building and maintaining the evaluation process for an AI assistant. The role centers on measuring performance, detecting regressions early, and ensuring evaluation results influence release decisions.

You will own the evaluation suite, implement checks to reveal quality changes over time, and ensure regressions are caught before customers are affected. This position emphasizes AI evaluation and release gating within a fast-paced product environment.