Staff Embedded Validation & Tools Engineer - OTA

ALSO.; Palo Alto (CA); On-site; USD 170,000 - 210,000; Full time

Benefits offered by this job

Health coverage

One Medical membership

Progyny fertility benefits

Job summary

ALSO. is seeking a Staff Embedded Validation & Tools Engineer - OTA to own validation for firmware and software update delivery across our fleet of embedded devices.

You will design and run validation programs to ensure updates are robust, secure, and reliable in real-world conditions, collaborating with firmware, release, QA, and cloud teams. You will build scalable testbeds, automate end-to-end OTA pipelines, and rigorously test bootloaders, agents, and update formats, driving rapid issue

Qualifications

• Bachelor's degree in Electrical Engineering, Computer Science, or related field.
• 7+ years validating embedded devices and firmware updates (OTA).
• Strong Python and C/C++ skills for embedded debugging.
• CI pipelines experience (Jenkins, GitLab CI, CircleCI) and test automation.

Responsibilities

• Design, implement, and execute OTA validation plans across device families and release types.
• Build, manage, and scale automated testbeds and harnesses for regression, integration, and stress testing.
• Automate end-to-end OTA pipelines, including signing, staging, rollout, and post-update verification.
• Validate update mechanisms and components (bootloader, update agent, partitions, recovery modes).
• Test resilience under intermittent networks, high latency, power interruptions, and storage constraints.
• Analyze logs, packet captures, and telemetry to diagnose failures; perform root cause analysis.

Skills

Python

C/C++

Embedded Linux

CI pipelines

HW debugging

Networking protocols

Debugging & problem-solving

Communication skills

Test automation

Jenkins/GitLab CI

Education

Bachelor's degree in Electrical Engineering or Computer Science

Tools

JTAG

Oscilloscopes

Logic analyzers

Wireshark

QEMU

Docker

Job description

As an Staff Embedded Validation & Tools Engineer - OTA, you'll own validation, automation, and operational readiness for firmware and software update delivery across our fleet of embedded devices. You'll design and run validation test programs that ensure OTA updates are robust, secure, and reliable under real-world conditions, including intermittent networks, power faults, device restarts, and other edge cases that matter to customers. You'll work closely with Firmware, Release, QA, and Cloud teams to close the loop from release to fleet health.

What You'll Do

• Design, implement, and execute comprehensive OTA validation plans across device families and release types (full OS images, firmware, bootloader, A/B updates, delta/patch updates)
• Build, manage, and scale automated testbeds and harnesses (hardware-in-the-loop, virtualized devices, emulators) to run regression, integration, and stress tests for OTA flows
• Implement automation for end-to-end OTA pipelines, from build artifact signing and staging to rollout and post-update verification, using Python, shell, and CI tools
• Validate update mechanisms and components, including the bootloader, update agent, package formats, rollbacks, A/B partitions, and recovery modes
• Test update resilience under adverse conditions such as intermittent networks (cellular/Wi-Fi), high latency/packet loss, power interruptions, low-battery scenarios, storage constraints, and corrupted images
• Validate security properties, including signing/verification, secure transport (TLS), secure boot, key management, and the vulnerability surface introduced by update components
• Define metrics and KPIs (success rate, rollback rate, mean time to recover, time-to-update), and instrument devices and cloud backends to measure rollout health and produce release readiness reports
• Perform root cause analysis on failed updates, reproduce field issues, and drive remediation with firmware and backend engineering teams
• Maintain device lab hardware (JTAG, serial consoles, network emulators, power controllers, logic analyzers) and tooling stacks (Wireshark, serial monitors, provisioning fixtures)
• Contribute to OTA process improvements, including staging strategies, canary rollouts, rollback thresholds, risk assessment templates, and runbook documentation

What You'll Bring

• Bachelor's degree in Electrical Engineering, Computer Science, or related field (or equivalent experience)
• 7+ years of hands-on experience validating embedded devices and firmware updates (OTA or similar)
• Strong programming/scripting ability in Python and familiarity with C/C++ for embedded debugging
• Deep understanding of embedded platforms (embedded Linux and/or RTOS), bootloaders (e.g., U-Boot), partition schemes, and A/B update models
• Practical experience building automated test harnesses and integrating tests into CI pipelines (Jenkins, GitLab CI, CircleCI, etc.)
• Experience with hardware debugging tools and workflows, including serial/UART, JTAG/SWD, logic analyzers, oscilloscopes, and power cycling rigs
• Familiarity with networking protocols used for OTA (HTTP(S), MQTT, CoAP) and wireless technologies (Wi-Fi, Ethernet, cellular, BLE), and experience analyzing logs, packet captures, and device telemetry to diagnose update failures
• Strong debugging skills and methodical problem solving, able to reproduce complex failure modes and propose design fixes, with excellent communication skills to write clear test plans, runbooks, and postmortems
• Preferred: Experience with cloud OTA platforms (AWS IoT Device Management, Mender, Azure IoT, Google Cloud IoT, or custom fleet management systems), secure update practices (code signing, PKI, TPM/secure element integration, secure boot), or CAN/LIN and automotive-grade validation
• Preferred: Familiarity with containerization/virtualization for test environments (Docker, QEMU), large-scale rollout strategies (canary/capacity ramping, staged rollouts, feature flags), firmware build systems (Yocto/OpenEmbedded, Buildroot), or telemetry frameworks and analytics (Prometheus, Grafana, ELK)

The salary for this position ranges from $170,000 - $210,000 per year, depending on experience, qualifications, and location.

• Robust health coverage — excellent health, dental and vision insurance covered up to 100% by ALSO with FSA & HSA options
• One Medical membership and dedicated insurance advocates
• Rich fertility and family building benefits with Progyny