Dev Ops Engineer

Caltech; Pasadena (CA); Hybrid; USD 109,000 - 135,000; Full time

Job summary

Caltech seeks an Associate DevOps Engineer to support the Deep Synoptic Array (DSA) project. You will help design, deploy, and manage on-prem infrastructure for processing and monitoring pipelines, with strong collaboration with the software group.

The role allows campus-based, Owens Valley, Ely, NV, or hybrid/remote work arrangements. You will lead provisioning automation, cluster management (Proxmox, Kubernetes, Slurm), and build out CI pipelines, while ensuring security and performance across

Qualifications

• Bachelor's degree or equivalent practical experience.
• 5+ years of professional DevOps or systems infrastructure experience.
• Extensive Linux server administration experience.
• Deep IaC experience (Terraform, Ansible, Puppet, or Chef).
• Containerization with Docker and orchestration with Kubernetes.
• CI pipelines using Jenkins or GitLab CI/CD.
• Scripting skills in Bash and Python.
• Experience with bare-metal provisioning tools like MaaS, Foreman, or xCAT.
• Strong networking knowledge for control and data traffic.
• Excellent communication and collaboration abilities.

Responsibilities

• Co-lead design, deployment, and management of on-prem infra for DSA pipelines.
• Develop automated server provisioning and config management processes.
• Maintain Proxmox, Kubernetes, and Slurm clusters for deployments.
• Configure routers/switches for control and high-bandwidth traffic.
• Package DSA software for various targets using UV, Cargo, and CMake; use Nix/Docker.
• Design and implement robust CI pipelines for rapid feedback.
• Create deployment configurations with Kubernetes manifests and Ansible playbooks.
• Establish comprehensive monitoring and logging of infrastructure health.
• Support security compliance with applicable standards.
• Collaborate with Software Development to understand needs and provide support.

Skills

Scripting (Bash, Python)

Linux administration

Communication skills

Collaboration

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Terraform

Ansible

Puppet

Chef

Foreman

MaaS

xCAT

Docker

Kubernetes

Jenkins

GitLab CI/CD

Proxmox

Slurm

UV

Cargo

CMake

Nix

Job description

Caltech is a world-renowned science and engineering institute that marshals some of the world's brightest minds and most innovative tools to address fundamental scientific questions. We thrive on finding and cultivating talented people who are passionated about what they do. Join us and be a part of the diverse Caltech community.

Job Summary

We are seeking a DevOps Engineer with experience managing large-scale, on-premises infrastructure deployments, to work on the Deep Synoptic Array (DSA), a world-leading radio telescope that is expected to commence construction in 2027. The array will use 1,650, 6.15-meter dishes to observe at radio wavelengths (0.7 - 2 GHz frequency range) to survey the skies 10x faster than any radio telescope current or planned. The telescope will conduct a broad range of research, including the study of the formation of galaxies, the search for cosmic explosions, and investigations into the nature of gravity.

The applicant would fulfill the role of Associate DevOps Engineer for the DSA project, working closely with and supporting the lead DevOps Engineer on high-impact infrastructure initiatives. This role will support the DSA Software Group as we progress through the final design stage to construction.

The role can be based at either the Caltech campus, the Owens Valley Radio Observatory in Big Pine, CA, Ely, NV, or hybrid/remote.

Essential Job Duties

• On-Premises Infrastructure Management: Co-lead the design, deployment, and management of the on-premises infrastructure for the DSA's processing and monitoring pipelines. This will involve provisioning and configuring hundreds to thousands of hosts with a variety of capabilities, including high performance GPU computing — whether through thousands of commercial GPUs or a handful of Nvidia NVL72 racks — and specialized signal processing hardware.
• Automation: Develop and implement automated processes for server provisioning, configuration management, and software testing, using tools like Ansible, Puppet, and Chef for configuration management alongside platforms like Foreman or MaaS.
• Clustering: Establish, maintain, and administer software deployment clusters like Proxmox, Kubernetes, and Slurm to provide high-level deployment targets for DSA software infrastructure.
• Networking: Maintain and optimize the configuration of routers and switches to support both the control traffic and high-bandwidth signal traffic of the DSA telescope.
• Packaging: Package DSA software for deployment to a variety of targets, including our production hardware and various testing platforms, using build tooling like UV, Cargo, and CMake alongside packaging and containerization solutions like Nix and Docker.
• CI Pipelines: Design and implement robust CI pipelines for the DSA software to ensure rapid feedback and consistent testing practices for proposed software changes.
• Deployment Orchestration: Develop and maintain deployment configurations for the DSA software packages, including Kubernetes manifests, Slurm jobs, and Ansible playbooks to ensure efficient deployments of updated DSA software.
• Monitoring and Logging: Establish and maintain comprehensive monitoring and logging systems to track infrastructure health and application performance.
• Security and Compliance: Support efforts to ensure compliance with relevant security standards and regulations.
• Collaboration: Collaborate closely with the Software Development team to understand their needs and provide ongoing support.

Basic Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience.
• 5+ years of professional experience in a DevOps, systems administration, or software infrastructure role.
• Extensive experience with Linux server administration.
• Deep experience with infrastructure as code (IaC) tools such as Terraform, Ansible, Puppet, or Chef.
• Proven experience with containerization technologies like Docker and orchestration platforms like Kubernetes.
• Experience designing and maintaining CI pipelines using tools like Jenkins, GitLab CI/CD, or similar solutions.
• Strong scripting and automation skills in Bash and Python.
• Experience with bare-metal provisioning tools such as MaaS, Foreman, or xCAT.
• Solid working knowledge of networking fundamentals (routers, switches) for control and data traffic.
• Strong communication and collaboration skills, with the ability to work independently and partner closely with a lead DevOps Engineer.

Preferred Qualifications

• Experience with clustering or job-scheduling platforms such as Proxmox or Slurm.
• Experience with build or packaging tools such as UV, Cargo, CMake, or Nix.
• Working knowledge of monitoring and logging tools like Prometheus, Grafana, or the ELK Stack.
• Experience with high-performance computing (HPC) systems.
• Working knowledge of timeseries databases like Prometheus or InfluxDB.

Required Documents

• Resume.
• Cover Letter.

Hiring Range

$109,200 - $134,700 Per Year

The salary of the finalist(s) selected for this role will be set based on a variety of factors, including but not limited to, internal equity, experience, education, specialty and training.

As one of the largest employers in Pasadena, CA, Caltech is committed to providing comprehensive benefits to eligible employees and their eligible dependents. Our benefits package includes competitive compensation, health, dental, and vision insurance, retirement savings plans, generous paid time off (vacation, holidays, sick time, parental leave, bereavement, etc.), tuition reimbursement, and more. Non-benefit eligible employees will have access to some benefits such as onsite counseling and sick time.

EEO Statement

We are an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to age, race, color, religion, sex, sexual orientation, gender identity, or national origin, disability status, protected veteran status, or any other characteristic protected by law.

Caltech is a VEVRAA Federal Contractor.

To read more Equal Employment Opportunity (EEO) go to eeoc_self_print_poster.pdf.

Disability Accommodations

Caltech complies with the Fair Employment and Housing Act (FEHA) and the Americans with Disabilities Act (ADA). We consider reasonable accommodation measures that may be necessary for eligible applicants and employees to perform the essential functions of a position.