AI Trust & Reliability Evaluation Scientist

HushOne, Inc.; Kirkland (WA); Hybrid; USD 120,000 - 170,000; Full time

Benefits offered by this job

Stock options

401(k) with match

Health, dental & vision

Job summary

HushOne, Inc. is seeking a role focused on rigorous AI evaluation design and measurement. You will craft evaluations using real workflows, adversarial inputs and distribution shifts, with blinded reviews and calibrated human evaluation where appropriate.

In the first 90 days you will deliver a versioned evaluation suite and a release report highlighting strengths and unresolved failures for easy inspection.

Qualifications

• Experimental design, statistics, and hands-on AI evaluation
• You prevent benchmark contamination deliberately, with a method you can describe
• You do not substitute a model judge for ground truth, and can explain when a judge is admissible
• You report failure rates as plainly as success rates
• Nice to have: Red-teaming or adversarial evaluation
• Nice to have: Human evaluation study design
• Nice to have: An eval suite that changed a shipping decision

Responsibilities

• Design evaluations for real workflows, adversarial inputs and distribution shifts
• Use held-out tasks, blinded review where appropriate and calibrated human evaluation
• Measure false completion, unauthorized actions, appropriate escalation and sustained usefulness over time
• Deliver a versioned evaluation suite and a release report in the first 90 days

Skills

Experimental design

Statistics

Hands-on AI evaluation

Adversarial evaluation (nice to have)

Human evaluation study design (nice to

Education

Educational background (not specified)

Tools

Statistical tools

Job description

HushOne, Inc. is seeking a role focused on rigorous AI evaluation design and measurement. You will craft evaluations using real workflows, adversarial inputs and distribution shifts, with blinded reviews and calibrated human evaluation where appropriate.

In the first 90 days you will deliver a versioned evaluation suite and a release report highlighting strengths and unresolved failures for easy inspection.