Remote-Ready ML Training Infrastructure Engineer

HushOne, Inc.; Kirkland (WA); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Stock options

401(k) with company match

Health/dental/vision

Gym membership

Remote-friendly

AI tokens

Job summary

HushOne, Inc. seeks a senior engineer to build robust ML training systems, enabling distributed execution and reproducible experiments on local resources. You will ensure memory accounting, numerical stability, and a clear boundary between local and remote workloads.

You will plan recoverable training runs, coordinate local inference with training, and implement checkpointing with attention to data permissions and safe preemption. Join a remote-friendly team in a hands-on environment.

Qualifications

• Practical ML training systems experience and distributed computing fundamentals.
• Memory accounting and numerical stability in practice.
• Understand fine-tuning vs large-scale training and design for the first.
• Make runs reproducible, including parts people usually omit.
• Nice to have: parameter-efficient fine-tuning methods.
• Nice to have: checkpointing and preemption handling.
• Nice to have: federated or on-device training.

Responsibilities

• Plan and implement training pipelines, checkpoints, optimizer-state handling, and distributed execution.
• Coordinate local inference with background training and enforce dataset permissions.
• Ensure a recoverable local fine-tuning workflow and clear boundary with remote workloads.
• Plan runs that survive power interruptions while preserving baseline comparability.

Skills

ML training systems

distributed computing

memory accounting

numerical stability

reproducible runs

fine-tuning vs large-scale training

Tools

Checkpointing

Federated learning

On-device training

Job description

HushOne, Inc. seeks a senior engineer to build robust ML training systems, enabling distributed execution and reproducible experiments on local resources. You will ensure memory accounting, numerical stability, and a clear boundary between local and remote workloads.

You will plan recoverable training runs, coordinate local inference with training, and implement checkpointing with attention to data permissions and safe preemption. Join a remote-friendly team in a hands-on environment.