Benchmark Engineer: Performance & Energy Analytics

HushOne, Inc.; Kirkland (WA); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Stock options

401(k) with company match

Health insurance

Gym membership

Remote-friendly

Job summary

HushOne, Inc. is seeking an engineer to design experiments and benchmarks that reveal how AI systems perform across inference, retrieval, training and integrations. You will define representative workloads, record model version, quality, context, precision, memory and device conditions.

Within your first 90 days, you will publish a repeatable baseline and a reproducible comparison. You will document methodology so others can reproduce findings and report results even when they are not favorable.

Qualifications

• Experimental design, systems measurement, and clear technical writing in combination
• You can identify a cherry-picked workload and distinguish noise from a meaningful change
• You publish methodology alongside results, so somebody else can reproduce them
• The independence to report a result the company will not enjoy

Responsibilities

• Design a fair comparison between two devices when their fastest available model configurations have different quality
• Build representative task suites covering inference, retrieval, training and integrations
• Record model version, quality, context, precision, memory and device conditions
• Measure latency distributions, energy use, and task cost alongside throughput

Skills

Experimental design

Systems measurement

Technical writing

Identify cherry-picked workloads

Job description

HushOne, Inc. is seeking an engineer to design experiments and benchmarks that reveal how AI systems perform across inference, retrieval, training and integrations. You will define representative workloads, record model version, quality, context, precision, memory and device conditions.

Within your first 90 days, you will publish a repeatable baseline and a reproducible comparison. You will document methodology so others can reproduce findings and report results even when they are not favorable.