Distributed Systems Engineer

HushOne, Inc.; Kirkland (WA); Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Stock options

Annual bonus

Health insurance

401(k) with match

Gym membership

Job summary

HushOne, Inc. seeks a systems engineer to design and implement durable task state, queues, leases, and reconciliation across a distributed setup. You will craft failover that prevents duplicate authority and clearly separate control-plane replication from execution workers.

You will also emphasize handling partial failures, explaining the chosen consistency model to implementers and demonstrating real-world failure guarantees. This role values rigorous reasoning and practical, testable outcomes.

Qualifications

• Strong reasoning about consistency, retries, failure detection, and distributed transactions.
• Operated a system through real failures and can say what guarantees it provided.
• Design for partial failure as the normal case.
• Explain a consistency model to someone who will implement against it.

Responsibilities

• Build durable task state, queues, leases, fencing, cancellation and reconciliation.
• Design failover so a recovered replica cannot become a second conflicting authority.
• Differentiate control-plane replication from execution workers and expose uncertainty when external outcomes cannot be confirmed.

Skills

Consistency modeling

Retry logic

Failure detection

Distributed transactions

System design

Explainable models

Tools

Consensus protocols

Formal methods

Job description

One person's agent should work across many machines without ever losing track of who is in charge. You make delegation and recovery predictable, which is a distributed-systems problem with an unusual constraint: the authority boundary is a single human being, and it must hold even when parts of the system are lying or gone.

The work:

Build durable task state, queues, leases, fencing, cancellation and reconciliation. Design failover so a recovered replica cannot become a second conflicting authority. Distinguish control-plane replication from execution workers and expose uncertainty when external outcomes cannot be confirmed.

What good looks like:

In your first 90 days, demonstrate delegated work across two devices, interruption recovery and prevention of duplicate externally visible actions in the supported test cases.

Evidence we look for:

Bring strong reasoning about consistency, retries, failure detection and distributed transactions. Show a system you operated through real failures and the guarantees it actually provided.

Required:

• Strong reasoning about consistency, retries, failure detection, and distributed transactions
• You have operated a system through real failures and can say what guarantees it actually provided, as opposed to claimed
• You design for partial failure as the normal case
• You can explain a consistency model to somebody who will have to implement against it

Nice to have:

• Consensus protocols in production
• Formal methods or model checking
• Edge or intermittently connected systems

The exercise:

Trace a task whose worker finishes after its lease expires and whose external API response is lost.

How we work:

We work together in the office, five days a week, and you can be based at any of our garages: Kirkland Garage (Kirkland, WA); UAE Garage (Dubai, Dubai). We hire across the United States , India and the UAE. We are remote-friendly around family: if you need to work from home some days to look after the people you love, we arrange that with you one person at a time, and we encourage people to use it rather than tough it out.

Pay:

We publish what we pay. Indicative ranges by market and level are on our compensation page, and they are realistic going market rates rather than headline numbers. Wherever we hire we pay at least the local market rate, and for full-time roles our floor is a living wage, never the statutory minimum. Compensation is reviewed every year and on promotion.

Equity:

Every full-time teammate gets stock options. Four-year vesting with a one-year cliff, sized to role, level and impact, and confirmed in writing at offer. High performers earn refresh grants.

Bonus and commission:

An annual performance bonus tied to clear company and personal goals, indicatively 10 to 20 percent of base for non-sales roles. Customer-facing roles carry on-target earnings, typically a 50/50 split of base and variable, with uncapped commission and accelerators above quota.

Health and peace of mind:

Medical, dental and vision for you and your family, plus life and disability cover. We have chosen the highest plan tier available to us rather than the cheapest one that clears the bar, because the point of this is that you never have to think about it. The specific plan numbers are confirmed in your offer letter.

401(k) with company matching:

A 401(k) with a company match, so the years you spend here compound into something that is yours whatever happens next. The match formula is confirmed in your offer letter.

AI tokens:

A budget of AI tokens of your own, because a company that says you should own your AI cannot be the company that rations it. Use them on the work and on whatever you are curious about.

Gym, and the everyday things:

A gym membership, and a corporate benefits programme with its own app, where you redeem real discounts with a long list of retailers on the ordinary purchases of a life.

Family:

We are remote-friendly around your family, arranged one person at a time, and we would rather you took it than toughed it out.

Referrals:

Refer someone we hire full-time who stays a year and you get $1,000 plus $10,000 in referral stock-based equity, on top of your own package.