Research Computing Systems Engineer

Stanford University; Palo Alto (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Stanford University seeks a talented systems engineer to join Stanford Research Computing, supporting a multi-petabyte, multi-platform research computing environment. You will steward bastion and backbone services, manage large-scale infrastructure, and collaborate with researchers across Stanford’s schools to enable advanced computing and data tools.

You will drive security, resilience, and operational excellence, including coordinating with vendors and internal partners to ensure reliable

Qualifications

• Bachelor's degree and ten years of relevant experience.

Responsibilities

• Lead the design, development, installation and maintenance of operating systems, utilities, and applications software on computing systems.
• Anticipate risks, de-escalate issues, and prevent emergencies to limit disruptions to system operations and protect data integrity.
• Safeguard university data and system assets and develop security policies and procedures for university IT systems.
• Establish and enforce systems policies ensuring compliance with external best practices.
• Collaborate with other IT specialties to align IT strategies and ensure technical feasibility of initiatives.
• Review capacity planning for system configurations, software services, and network interdependencies.
• Act as technical lead for university-wide computer system administration and potentially manage staff.
• Provide project management for large and complex university-wide computing projects.
• Manage vendor relationships and negotiate hardware and software maintenance agreements.

Job description

Stanford Research Computing is looking for a talented systems engineer to join our team of collaborative and innovative professionals helping Stanford's faculty and students use advanced computing and data tools to explore new frontiers in knowledge and solve some of humanity's most urgent problems. Our staff work directly with some of the world's top researchers in a broad range of disciplines, across all of Stanford's seven schools — while also supporting and learning from each other in cross-project endeavors. We maintain and steadily improve an advanced research computing facility, and we support a variety of environments for Stanford research. In Stanford Research Computing, you'll have a rare opportunity to contribute to discoveries and inventions that have global reach and positive impact, and to share in the curiosity and commitment of the scholars and scientists who lead these projects.

About the Role

Stanford Research Computing operates a portfolio of large-scale HPC clusters and petabyte-scale storage systems serving thousands of researchers. Every one of those platforms depends on a layer of shared infrastructure that researchers never see: the bastion hosts our administrators pass through to reach management networks, the network license servers that let research software start, the container hosts carrying our internal services, and the observability stack that tells us something is wrong before a researcher has to.

In this role you will be the primary steward of that layer, serving as lead for the "backbone" services the rest of our environment depends on. You will also contribute to a co-location initiative, standing up management infrastructure and geographically distinct replication in a remote data center. The work spans a wide range of technologies, with substantial engineering ahead in hardware lifecycle, redundancy, and configuration management.

Why Stanford?

You won't be maintaining back-office IT. You will be building the connective tissue that a multi-petabyte, multi-platform research computing environment runs on — the systems that decide whether a downtime is a scheduled inconvenience or a lost quarter of someone's research.

Responsibilities

• Bastion hosts: Lead the administration, hardening, and modernization of bastion and jump hosts across multiple data centers, and design the next generation with redundancy and high availability. Support partner unit administrators moving from VDI to bastion-based access.
• Network license services: Operate and improve the primary and secondary FLEXlm license servers behind commercial research and statistical software. Right-size firewall policy, retire orphaned license hosts, and bring runbooks and user documentation current.
• Central observability service: Lead the central observability service and the servers behind it — Prometheus, Grafana, Splunk, XDMoD, and facility telemetry. Consolidate legacy monitoring and build alerting that reaches the right person with enough context to act.
• Co-location project: Support a funded co-location initiative from design through production hand-off: network architecture, management infrastructure and a bastion host in the remote facility, and a geographically distinct replication target for independent backups.
• Container hosts and backbone services: Lead the container hosts and the internal services they carry, along with cluster head and service nodes, bare-metal provisioning, and the VMs behind other Research Computing backbone services. Bring these under Ansible and Git-based configuration management.
• Security and compliance: Establish and maintain secure build standards across the systems you own, in line with university, regulatory, and contractual requirements. Deploy endpoint and logging agents, remediate scan findings, and manage credential and access lifecycle.
• Resilience and documentation: Reduce single points of failure, and document core service requirements and details so that any member of the team can operate, restore, or rebuild a service.
• Planning and hardware lifecycle: Meet regularly with platform leads and service owners to understand upcoming needs, own hardware lifecycle and capital planning for the infrastructure you lead, and help develop business cases for replacement and expansion.
• Vendor engagement: Liaise with hardware and software vendors and support partners to triage and resolve issues, manage RMAs, and keep systems under appropriate warranty and support coverage.

Responsibilities

Core Duties:

• Lead the design, development, installation and maintenance ofoperating systems, utilities, and applications software on computingsystems.

• Anticipate risks, de-escalate issues, and prevent emergencies tolimit disruptions to system operations and protect the integrity ofuserdata and systems.

• Safeguard the university’s data and system assets – formulatesystem security strategies and develop viable policies and proceduresthat will enable the design and implementation of system securitymeasures at the university.

• Establish and enforce systems policies and procedures andvalidate that university software/hardware standards are aligned withexternal best practice.

• Partner with other information technology specialty areas toconfirm information technology strategies, devise and deploy plans toensure information technology objectives are met, and advise ontechnical feasibility of information technology initiatives,particularly regarding system compatibility within the university’scurrent, or proposed technical or structural framework(s).

• Review and conduct capacity planning for system configuration,software services, network services, load distribution, and serviceinterrelationships among computer systems.

• Act as technical expert or lead for university-wide computersystem administration. May manage system administration staff.

• Provide project management for large and complex university-widecomputing projects.

• Manage vendor relationships and negotiate cost effective hardwareand software maintenance agreements with vendors.

Minimum Education and Experience:

Bachelor's degree and ten years of relevant experience, or acombination of education and relevant experience.

Knowledge, Skills and Abilities:

• Extensive experience with complex, multi-system platforms andvendors.

• Notable experience coordinating multi-system and computingenvironments in independent computing facilities.

• Extensive experience developing/implementing a businesscontinuation and disaster recovery plan.

• Exceptional ability to develop appropriate plans to meetcomputing needs.

• Expert ability to program in multiple programming languages inmultiple operating systems.

• Superior ability to lead and work on large/complex systemdeployment projects in a team environment.

• Expert knowledge of security trends and best practice.