AI Evaluation and Reliability Scientist

HushOne, Inc.; Kirkland (WA); Hybrid; USD 120,000 - 170,000; Full time

Benefits offered by this job

Stock options

401(k) with match

Health, dental & vision

Job summary

HushOne, Inc. is seeking a role focused on rigorous AI evaluation design and measurement. You will craft evaluations using real workflows, adversarial inputs and distribution shifts, with blinded reviews and calibrated human evaluation where appropriate.

In the first 90 days you will deliver a versioned evaluation suite and a release report highlighting strengths and unresolved failures for easy inspection.

Qualifications

• Experimental design, statistics, and hands-on AI evaluation
• You prevent benchmark contamination deliberately, with a method you can describe
• You do not substitute a model judge for ground truth, and can explain when a judge is admissible
• You report failure rates as plainly as success rates
• Nice to have: Red-teaming or adversarial evaluation
• Nice to have: Human evaluation study design
• Nice to have: An eval suite that changed a shipping decision

Responsibilities

• Design evaluations for real workflows, adversarial inputs and distribution shifts
• Use held-out tasks, blinded review where appropriate and calibrated human evaluation
• Measure false completion, unauthorized actions, appropriate escalation and sustained usefulness over time
• Deliver a versioned evaluation suite and a release report in the first 90 days

Skills

Experimental design

Statistics

Hands-on AI evaluation

Adversarial evaluation (nice to have)

Human evaluation study design (nice to

Education

Educational background (not specified)

Tools

Statistical tools

Job description

You show when a private agent deserves a user's trust, building the evidence about task completion, failure, and human oversight. Two failure modes govern this role: benchmark contamination, and quietly substituting a model judge for ground truth because it is cheaper.

The work: Design evaluations for real workflows, adversarial inputs and distribution shifts. Use held-out tasks, blinded review where appropriate and calibrated human evaluation. Measure false completion, unauthorized actions, appropriate escalation and sustained usefulness over time.

What good looks like: In your first 90 days, deliver a versioned evaluation suite and a release report that makes strengths and unresolved failures easy to inspect.

Evidence we look for: Bring experimental design, statistics and hands-on AI evaluation. Show how you prevent benchmark contamination and avoid substituting a model judge for ground truth.

Required:

• Experimental design, statistics, and hands-on AI evaluation
• You prevent benchmark contamination deliberately, with a method you can describe
• You do not substitute a model judge for ground truth, and can explain when a judge is admissible
• You report failure rates as plainly as success rates

Nice to have:

• Red-teaming or adversarial evaluation
• Human evaluation study design
• You have built an eval suite that changed a shipping decision

The exercise: Design an evaluation that detects an agent becoming more persuasive without becoming more accurate or reliable.

How we work: We work together in the office, five days a week, and you can be based at any of our garages: Kirkland Garage (Kirkland, WA); UAE Garage (Dubai, Dubai). We hire across the United States, India and the UAE. We are remote-friendly around family: if you need to work from home some days to look after the people you love, we arrange that with you one person at a time, and we encourage people to use it rather than tough it out.

Pay: We publish what we pay. Indicative ranges by market and level are on our compensation page, and they are realistic going market rates rather than headline numbers. Whenever we hire we pay at least the local market rate, and for full-time roles our floor is a living wage, never the statutory minimum. Compensation is reviewed every year and on promotion.

Equity: Every full-time teammate gets stock options. Four-year vesting with a one-year cliff, sized to role, level and impact, and confirmed in writing at offer. High performers earn refresh grants.

Bonus and commission: An annual performance bonus tied to clear company and personal goals, indicatively 10 to 20 percent of base for non-sales roles. Customer-facing roles carry on-target earnings, typically a 50/50 split of base and variable, with uncapped commission and accelerators above quota.

Health and peace of mind: Medical, dental and vision for you and your family, plus life and disability cover. We have chosen the highest plan tier available to us rather than the cheapest one that clears the bar, because the point of this is that you never have to think about it. The specific plan numbers are confirmed in your offer letter.

401(k) with company matching: A 401(k) with a company match, so the years you spend here compound into something that is yours whatever happens next. The match formula is confirmed in your offer letter.

AI tokens: A budget of AI tokens of your own, because a company that says you should own your AI cannot be the company that rations it. Use them on the work and on whatever you are curious about.

Gym, and the everyday things: A gym membership, and a corporate benefits programme with its own app, where you redeem real discounts with a long list of retailers on the ordinary purchases of a life.

Family: We are remote-friendly around your family, arranged one person at a time, and we would rather you took it than toughed it out.

Referrals: Refer someone we hire full-time who stays a year and you get $1,000 plus $10,000 in referral stock-based equity, on top of your own package.