Staff OTA Systems Engineer — Remote

Horizon3; Oregon; Hybrid; USD 230,000 - 260,000; Full time

Benefits offered by this job

Health, vision & dental care

Flexible vacation policy

Parental leave

Job summary

Horizon3 is seeking an Expert Engineer for the OTA Services team to anchor how NodeZero updates itself across thousands of deployments — safely, without downtime, and without introducing new vulnerabilities.

You will write production code, shape architecture for update delivery and version management, and mentor P4 engineers, reporting to the Client Platform Engineering Manager. Expect a challenging, scalable systems role in a fully remote company.

Qualifications

• 8+ years of professional software engineering experience with deep specialization in distributed systems, remote update systems, or fleet management infrastructure.
• Deep understanding of remote update system design — delivery protocols, version negotiation, delta delivery, health verification, and rollback semantics.
• Experience designing rollout strategies for software deployed across heterogeneous, customer-managed environments.
• Strong proficiency in Go, Python, or Rust for systems-level infrastructure work.
• Experience operating in constrained or air-gapped environments and designing delivery systems that don’t assume perfect connectivity.
• Deep understanding of Kubernetes — deployments, updates, and monitoring in running clusters.
• Experience with container image distribution: OCI artifacts, image manifests, and registry protocol.
• Understanding of mTLS, certificate lifecycle, and secure channel establishment for update delivery.
• Track record of engineering leadership and setting standards for reliability, testing, and architecture documents.
• Experience mentoring mid-level engineers through design review and code review.

Responsibilities

• Design and own the over-the-air update system — the delivery protocol, version negotiation, rollout orchestration, health verification, and automatic rollback.
• Own the systems that track the version state of every NodeZero deployment — know what's running where, what's pending, and what's stuck.
• Design safe rollout strategies for updates — phased rollouts, canary deployments, customer pinning, and emergency hold mechanisms.
• Design delivery mechanisms that work reliably across diverse customer environments including air-gapped networks and strict egress policies.
• Own OTA system reliability end-to-end — on-call for delivery failures, stuck updates, and version drift incidents.
• Mentor engineers through design review, code review, and technical pairing; be the go-to person for complex delivery semantics.

Skills

Distributed systems

Delivery systems

Go

Python

Rust

Kubernetes

mTLS

Architecture design

Mentoring

Tools

OCI artifacts

Image registries

Container images

Job description

Horizon3 is seeking an Expert Engineer for the OTA Services team to anchor how NodeZero updates itself across thousands of deployments — safely, without downtime, and without introducing new vulnerabilities.

You will write production code, shape architecture for update delivery and version management, and mentor P4 engineers, reporting to the Client Platform Engineering Manager. Expect a challenging, scalable systems role in a fully remote company.