Senior Incident Leader - Reliability & Observability

Robinhood; California (MO); On-site; USD 196,000 - 230,000; Full time

Benefits offered by this job

Health insurance

Equity ownership

401(k) matching

Lifestyle wallet

Life & disability insurance

Catered meals

Job summary

Robinhood is seeking a Senior Engineer for the newly formed Robinhood Command Center (RCC) in Menlo Park, CA. You will lead incident response, drive reliability and observability initiatives, and own tooling and processes for rapid, high-quality incident handling.

You will collaborate across product engineering, reliability, observability, and infrastructure teams to reduce customer impact and improve MTTD/MTTR.

Qualifications

• 5+ years of software engineering experience and operating production systems.
• 2+ years in reliability engineering, infrastructure, distributed systems, or production operations.
• Hands-on incident leadership in roles like IMOC or incident commander.
• Strong communication and cross-functional collaboration during high-severity incidents.
• Deep knowledge of observability, fault-tolerance, and reliability patterns.

Responsibilities

• Lead long-term reliability and observability strategy across Robinhood's infrastructure.
• Collaborate with engineers to raise operational excellence and incident response.
• Coordinate incident mitigation, rollbacks, and traffic shifts during incidents.
• Develop incident management processes to minimize customer impact.
• Own global dashboards and alerts tied to CUJs and business metrics.
• Evolve incident response tooling, education, adoption, and MTTR/MTTD improvements.
• Drive post-incident governance with postmortems and SEV reviews.
• Design failure mitigation strategies to avoid full-region/datacenter failovers.
• Build frameworks to improve monitoring and observability across services.
• Own roadmap to bring observability to critical user journeys.
• Deliver insights to executives about service quality and reliability.
• Mentor and influence engineering culture and hiring.

Skills

Software engineering

Reliability engineering

Incident leadership

Cross-functional collaboration

Fault-tolerant design

Multi-region architectures

Tools

OpenTelemetry

Prometheus

Grafana

Job description

Robinhood is seeking a Senior Engineer for the newly formed Robinhood Command Center (RCC) in Menlo Park, CA. You will lead incident response, drive reliability and observability initiatives, and own tooling and processes for rapid, high-quality incident handling.

You will collaborate across product engineering, reliability, observability, and infrastructure teams to reduce customer impact and improve MTTD/MTTR.