Senior Software Engineer, Resilience Engineering - DGX Cloud

NVIDIA; Santa Clara (CA); On-site; USD 184,000 - 356,500; Full time

Benefits offered by this job

Equity options

Comprehensive benefits

Job summary

NVIDIA is seeking a seasoned professional to drive its reliability strategies, focusing on operational excellence in a 24/7 environment. The role requires a hands-on leader with 10+ years of experience and a strong software engineering background, particularly in languages like Go and Python.

The successful candidate will lead incident responses and implement modern reliability practices such as chaos engineering. Compensation includes a competitive salary range from $184,000 to $356,500 based on level and experience, along with equity options and benefits.

Qualifications

• 10+ years of industry experience with large-scale production systems.
• Proven track record in operational reliability and SLO establishment.
• Hands-on experience in modern observability tools.

Responsibilities

• Build and strategize reliability practices across the organization.
• Lead high-severity incident response efforts.
• Implement chaos engineering and resilience testing.

Skills

Software engineering skills (Go, Python)

Experience with SLO programs

Reliability engineering practices

Cascading dependencies understanding

Influencing across team boundaries

Education

Bachelor’s or Master’s degree

Tools

Prometheus

OpenTelemetry

Grafana

PagerDuty

Rootly

Job description

Responsibilities

• Build org‑wide reliability strategy, guiding how NVIDIA matures its operational practices in a 24/7 environment.
• Stand up a rigorous SLO program, defining and maintaining high standards across teams.
• Lead incident response for high‑severity incidents, ensuring low drama and high‑signal resolution.
• Build and improve production code daily, enhancing our data platform and related tooling.
• Implement chaos engineering, failure injection, and resilience testing to elevate the team’s standard practices.
• Improve standards by setting an example with your hands‑on experience and leadership.

Qualifications

• Deep, hands‑on experience running large‑scale production systems with a proven track record.
• Detailed understanding of failure modes in large systems, including cascading dependencies and retry storms.
• Strong software engineering skills with current, hands‑on experience in Go, Python, or similar languages.
• Proven experience establishing and maintaining an SLO program with operational rigor.
• Practical experience in reliability fields such as chaos engineering and failure injection.
• Ability to influence across team boundaries through credibility and expertise.
• 10+ years of industry experience with a Bachelor’s or Master’s degree, or equivalent experience operating systems at scale.

Ways to Stand Out

• Experience within a world‑class reliability function like Google SRE or Meta production engineering.
• Expertise in operating GPU, HPC, or AI training infrastructure with outstanding failure modes.
• A track record of measurable reliability improvements within an organization.
• Proficiency with modern observability and operational tools such as Prometheus, OpenTelemetry, Grafana, PagerDuty, and Rootly.

Benefits & Compensation

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions.

Base salary range: Level 4 - 184,000 USD to 287,500 USD. Level 5 - 224,000 USD to 356,500 USD.

You will also be eligible for equity and benefits.

Equal Employment Opportunity Statement

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. We do not discriminate on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status, or any other characteristic protected by law.