Staff Triage & RCA Engineer

Agility Robotics; Fremont (CA); Hybrid; USD 150,000 - 190,000; Full time

Job summary

Agility Robotics in Fremont, CA seeks a Staff Triage & RCA Engineer to provide technical leadership for software issue triage and root cause analysis across our robotic systems. You will define processes, tooling, and diagnostic systems to diagnose and resolve failures at scale.

You will mentor engineers, establish standards for fault isolation, and drive observability improvements across hardware and software stacks, including logs, telemetry, and system behavior analyses.

Qualifications

• 10+ years in Systems Integration, software-hardware interfaces, or similar roles.

Responsibilities

• Lead software/firmware and system-level failure triage.
• Define standardized triage processes, workflows, and criteria.
• Lead RCA investigations and author high-quality reports.
• Mentor engineers and set technical direction for diagnostics tooling.

Skills

Triage leadership

Root cause analysis

Data analysis

Mentoring

System observability

Education

Bachelor's or Master’s in CS/Robotics

Tools

Datadog

Splunk

New Relic

Job description

Agility’s commercially deployed humanoids operate alongside teams in warehouses, manufacturing facilities, and distribution centers—tackling physically demanding and repetitive tasks while enabling workers to focus on higher-value work. With industry-leading safety standards and years of proven deployment data, we're pioneering a new era of automation that enhances human potential.

About the Role

Agility Robotics is at the forefront of humanoid robotics, redefining how robots interact with and navigate the human world. Our robots rely on a complex intersection of hardware and advanced software to perform tasks in dynamic environments.

We are seeking a Staff Triage & RCA Engineer to provide technical leadership for software issue triage and root cause analysis across our robotic systems. In addition to leading complex investigations, this role will define the processes, tooling, and technical systems used to diagnose and resolve failures at scale.

You will serve as a technical authority for ambiguous hardware-software failures, establish standards and methodologies for triage and RCA, and drive the development of diagnostic tools and infrastructure that improve the efficiency and effectiveness of the broader engineering organization. You will also mentor engineers and help set technical direction for the discipline.

About the Work

Issue Triage

• Serve as a technical lead for complex software, firmware, and system-level failure triage.
• Define and improve standardized triage processes, workflows, and disposition criteria.
• Lead prioritization and escalation of critical software-related failures, ensuring engineering teams receive clear, actionable investigations.
• Identify systemic failure trends across the fleet and drive appropriate corrective actions.

Root Cause Analysis

• Lead end-to-end investigations of novel and ambiguous failures at the hardware-software interface.
• Use logs, telemetry, video, and system behavior to identify definitive root causes and contributing factors.
• Establish and promote rigorous RCA methodologies and standards across the organization.
• Author high-quality RCA reports and troubleshooting guidance that enable broader engineering teams to independently investigate failures.

Tooling & Continuous Improvement

• Define the technical direction and roadmap for triage, diagnostics, and RCA tooling.
• Develop scalable diagnostic scripts, automation, and data analysis frameworks to accelerate investigations.
• Drive improvements to system observability, telemetry, logging, and diagnosability.
• Identify opportunities to automate recurring investigations and improve the overall engineering feedback loop.

Technical Leadership & Mentorship

• Serve as a technical subject matter expert and mentor for engineers performing triage and RCA.
• Establish technical standards and best practices for failure investigation and documentation.
• Influence Software, Hardware, Controls, and Systems teams on approaches to fault isolation, diagnosability, and system reliability.
• Provide technical direction for initiatives that improve the organization's ability to detect, diagnose, and resolve failures.

About You

Experience:

• 10+ years of experience in Systems Integration, Software-Hardware interface, R&D, or similar roles involving complex mechatronic or autonomous systems.
• Proven track record of leading complex, ambiguous technical investigations and defining scalable engineering processes or tooling.
• Experience with monitoring and observability platforms (e.g., Datadog, Splunk, or New Relic).
• Experience working with cloud storage and databases (e.g., AWS S3, SQL, or NoSQL) to retrieve and analyze large-scale telemetry and video datasets.
• Experience influencing cross-functional engineering teams and mentoring technical contributors.
• Preferred: Experience with HW/SW integration and design on HiL.

Technical Expertise:

• Mastery of log parsing and proficiency with Python or similar scripting languages for data analysis and automation.
• Strong understanding of robotics software stacks and communication protocols such as EtherCAT and CAN.
• Experience correlating video and/or hardware symptoms with system telemetry to identify software and system failures.
• Experience developing diagnostic, observability, or automation tooling for complex systems.
• Preferred: Experience troubleshooting HW/SW interactions involving cameras, encoders, IMUs, or other sensors.

Skills:

• Ability to define technical direction and solve ambiguous, unprecedented problems with scalable solutions.
• Ability to balance hands-on technical investigation with longer-term process, tooling, and systems development.
• Exceptional ability to synthesize complex data into clear, actionable insights.
• Strong technical leadership, communication, and mentoring skills.

Education:

Bachelor’s or Master’s degree in Computer Science, Robotics, Electrical Engineering, or a related field.