Site Reliability Engineer Senior

Northwestern University; Evanston (IL); Hybrid; USD 112,000 - 123,000; Full time

Job summary

Northwestern University’s Research Computing Infrastructure (RCI) seeks a Senior Site Reliability Engineer to lead design, implementation, and automation of HPC systems and researcher-facing services. You will develop software and automation to support HPC operations, maintain engineering standards across compute/storage/scheduling environments, and contribute to service reliability.

The role combines systems engineering, software development, and operations ownership with on-call rotations and

Qualifications

• Bachelor's degree or higher in computer science, information technology, or related field.
• 4 years of system engineering or equivalent experience required.
• Experience with Linux, monitoring tools (Grafana), version control (Git), and Python.

Responsibilities

• Leads planning, development, and coordination of RCI operations and projects.
• Provides recommendations on infrastructure, operations, automation, site reliability, and self-service enablement.
• Guides junior staff to deliver best practices in daily operations and code releases.
• Serves as a support escalation point for related projects and incidents.
• Develops solutions using HPC-focused tools and dashboards; participates in project meetings.

Skills

Linux

Python

Git

Bash scripting

Monitoring

Education

Bachelor's degree in computer science or IT

Tools

Grafana

Git

Job description

Department: IT Cyber Platform Services

Salary/Grade: ITS/82

Target hiring range for this position will be between $111,800-123,000 per year. Offered salary will be determined by the applicant's education, experience, knowledge, skills and abilities, as well as internal equity and alignment with market data

Job Summary

Research Computing Infrastructure's (RCI) Senior Site Reliability Engineer leads efforts and projects associated with the design, implementation, software development, automation, and reliable operation of Northwestern's high-performance computing (HPC) systems and research computing services.

As RCI's Senior Site Reliability Engineer, you will combine systems engineering, software development, application and service ownership, monitoring, and automation to continually improve Northwestern's evolving HPC environments. You will develop and maintain software and automation supporting HPC operations and researcher-facing services, while implementing and maintaining engineering standards across RCI's compute, storage, scheduling, monitoring, and supporting application environments. This role uses accepted software development, site reliability engineering, and IT practices to deliver reliable and scalable HPC services for Northwestern researchers.

This role offers a hybrid work schedule and takes part in regular 24/7 on-call HPC support rotations.

Specific Responsibilities

• Using standard systems management practices, leads planning, development, and coordination of RCI operations and projects.
• Provides recommendations regarding infrastructure, operations, automation, site reliability, and self-service enablement for systems and services.
• Provides guidance to junior staff to deliver best practices in daily operations and code releases.
• Serves as a support escalation point for all related projects and incidents.
• Installs or develops solutions through the understanding of requirements and attends project meetings. This will include an HPC user management application such as ColdFront and an in-house operational dashboard for HPC systems.
• Identifies training needs for self and coordinates the timeline with management.
• Keeps current on Information technology trends and development frameworks.
• Creates or expands solutions using advanced development frameworks and reviews code with team peers.
• Performs incident reviews after each P1/P2/P3, present to the team quarterly
• Ensures all services are appropriately monitored through regular reviews
• Creates new automations through cron jobs, Ansible notebooks, graphana and dashboards to measurably improve service metrics including compute hours delivered
• Responsible for RCI github repo and code reviews
• Assists with technical strategy and roadmap creation and updates.
• Recommends updates to operating system for use within HPC systems.
• Develops project timelines for on-going projects within the team.
• Troubleshoots HPC systems.
• Provides on-call support rotation; participation in service incidents and problem resolution efforts to support entrusted applications/products.
• Consults on platform best practices for users when appropriate.
• Provides data and metrics to support requirements for design and budget decisions.

Miscellaneous

Performs other duties as assigned.

Minimum Qualifications

• Successful completion of a full 4-year course of study in an accredited college or university leading to a bachelor's or higher degree in a major such as computer science, information technology, or related; OR appropriate combination of education and experience.
• 4 years of system engineering or equivalent experience required.
• Experience with Linux, monitoring tools such as Graphana, version control such as git, and Python

Minimum Competencies: (Skills, knowledge, and abilities.)

• Thorough knowledge of typical application architecture, infrastructure, and identity management technologies.
• Thorough knowledge of software development frameworks, infrastructure, and associated technologies.
• Experience with bash programming and thorough knowledge of procedural and object-oriented coding.

Benefits

At Northwestern, we are proud to provide meaningful and competitive benefits. The available benefits package for this position will include health, dental, vision, disability, and life insurance; paid vacation and holidays; paid medical/sick and parentalleave;tuition benefits for the employee and dependents; pre-tax and flex spending accounts for commuting and dependent care; generous retirement savings options; and wellness programs. For a comprehensive overview of available benefits, including eligibility details, visit us at https://www.northwestern.edu/hr/benefits/index.html to learn more.

Work-Life and Wellness

Northwestern offers comprehensive programs and services to help you and your family navigate life's challenges and opportunities, and adopt and maintain healthy lifestyles.

We support flexible work arrangements where possible and programs to help you locate and pay for quality, affordable childcare and senior/adult care. Visit us at https://www.northwestern.edu/hr/benefits/work-life/index.html to learn more.

Professional Growth & Development

Northwestern supports employee career development in all circumstances whether your workspace is on campus or at home. If you're interested in developing your professional potential or continuing your formal education, we offer a variety of tools and resources. Visit us at https://www.northwestern.edu/hr/learning/index.html to learn more.

Northwestern University is an Equal Opportunity Employer and does not discriminate on the basis of protected characteristics, including disability and veteran status. View Northwestern's non-discrimination statement. Job applicants who wish to request an accommodation in the application or hiring process should contact the Office of Civil Rights and Title IX Compliance. View additional information on the accommodations process.