Staff / Principal Systems Operations Engineer (SysOps)

Metrosion Inc.; Burlingame (CA); On-site; USD 180,000 - 290,000; Full time

Job summary

Metrosion Inc. is seeking a Staff/Principal Systems Operations Engineer in Burlingame, CA to own the production infrastructure for AI software. You will design, provision, and operate cloud services, implement IaC, and build robust CI/CD pipelines to enable reliable deployments.

You will lead incident response, improve runbooks, and collaborate with security for least-privilege access and disaster recovery planning. This is a hands-on, on-site role with strong leadership expectations.

Qualifications

• 8+ years in systems operations, SRE, platform engineering, DevOps, or production infrastructure.
• 12+ years at Principal level based on scope and impact.
• Experience at an AI-forward company is required.
• Leadership in architecture, reliability, or operational improvements across teams.

Responsibilities

• Design, provision, and operate cloud infrastructure, Linux systems, networking, storage, databases, and containerized services for AI products.
• Automate infrastructure as code and configuration management to ensure reproducibility.
• Develop CI/CD pipelines with testing, rollouts, and controlled production access.
• Define SLIs/SLOs and build observability with dashboards and alerts.
• Lead incident investigations, improve runbooks, and drive recovery improvements.
• Implement least-privilege access, secrets management, encryption, and patching in collaboration with security.
• Design backup, DR, and failover procedures and exercise them regularly.
• Operate AI workloads with attention to resource limits and external provider failures.
• Plan capacity, investigate performance and cost, and avoid bottlenecks.
• Maintain production docs and access/change evidence for audits.

Skills

Linux systems

Networking

Python

Go

Shell scripting

CI/CD pipelines

Observability

Incident response

Security engineering

Cloud platforms

Automation

Tools

Docker

Kubernetes

Terraform

CI/CD systems

Job description

Lead secure, reliable production infrastructure for AI software at Staff or Principal level, with hands‑on ownership of deployment, observability, resilience, and recovery.

The opportunity

Build the operational foundation for AI software used in regulated industries and critical operations. Partner with software engineering, data science, and security colleagues to turn product requirements into infrastructure that teams can deploy, understand, protect, and recover.

This is a hands‑on engineering role. Automate repeatable operations, make failure visible, and reduce the need for manual intervention. Treat access boundaries, customer information, change control, and operational evidence as part of the system design.

Staff Systems Operations Engineers lead the reliability and operational design of substantial services or platforms, establish practical deployment and recovery standards, and mentor colleagues. Principal Systems Operations Engineers set infrastructure strategy across products, resolve risks that span systems and teams, and guide long‑term architecture, capacity, and operational investment.

Staff level requires at least 8 years of relevant professional experience; Principal level requires at least 12 years. Demonstrated technical leadership, scope, judgment, and impact determine the level under consideration.

What you will do

• Design, provision, and operate cloud infrastructure, Linux systems, networking, storage, databases, and containerized services for AI products.
• Build reviewed, version‑controlled infrastructure as code and configuration automation; make environment changes reproducible and traceable.
• Develop CI/CD pipelines with appropriate testing, artifact integrity checks, staged rollout, rollback, and controlled access to production.
• Define service‑level objectives with product and engineering teams; build metrics, logs, traces, dashboards, and actionable alerts around user‑visible behavior.
• Lead incident investigation and recovery, maintain useful runbooks, and turn incident reviews into verified improvements to systems and operating practices.
• Implement least‑privilege access, network isolation, secrets management, encryption, patching, and vulnerability remediation in partnership with security colleagues.
• Design backup, restore, disaster recovery, and service failover procedures; regularly exercise them against agreed recovery objectives.
• Operate model‑serving and AI workflow dependencies with attention to resource limits, request queues, timeouts, rate limits, retries, and external‑provider failures.
• Plan capacity and investigate performance and infrastructure cost; identify bottlenecks before they affect critical workflows.
• Maintain operational documentation and evidence of access, configuration, and production changes to support customer and assurance reviews.
• Improve developer self‑service through reliable environments and automation; mentor engineers in production readiness and operational ownership.

What you bring

• Staff level: at least 8 years of relevant professional experience in systems operations, site reliability engineering, platform engineering, DevOps, or production infrastructure.
• Principal level: at least 12 years of relevant professional experience in systems operations, site reliability engineering, platform engineering, DevOps, or production infrastructure.
• Prior professional experience working at an AI‑forward company is required.
• A record of operating consequential production services and leading architecture, reliability, or operational improvements across teams.
• Deep hands‑on knowledge of Linux, networking, DNS, TLS, identity and access management, storage, and distributed‑system failure modes.
• Strong infrastructure automation skills using Python or Go and shell scripting, with maintainable code, tests, and review practices.
• Production experience with a major cloud platform, containers and orchestration, infrastructure as code, and CI/CD systems.
• Experience defining service‑level objectives, building observability, investigating incidents, and validating backup and recovery procedures.
• Practical security judgment around sensitive information, tenant isolation, secrets, patching, privileged access, and controlled production changes.
• Hands‑on understanding of AI workload operations, including inference dependencies, model endpoints, data pipelines, resource constraints, and model‑provider failures.
• Clear technical writing, calm incident communication, and evidence of mentoring colleagues and improving shared operating standards.

Success in this role

• Teams can deploy and recover services through documented, repeatable processes with clear ownership and appropriate controls.
• Monitoring reflects user outcomes, alerts lead to useful action, and incident learning reduces recurring failures.
• Recovery procedures are exercised, production access is accountable, and infrastructure changes have reviewable evidence.
• AI workloads meet agreed reliability, performance, and capacity expectations, with infrastructure cost and remaining operational risks visible.

Helpful experience

• Operating software in regulated industries or other settings with sensitive information, audit requirements, or strict availability expectations.
• GPU infrastructure, inference scheduling, model‑serving platforms, or optimization of AI workload capacity and cost.
• Multi‑region systems, fault injection, supply‑chain security, policy as code, or automated compliance evidence.
• Building shared infrastructure and operational practices for multiple products or a growing engineering organization.

Location & working arrangement

This is a full‑time position based in our Burlingame, California office and requires working on‑site five days a week.

AI & ML knowledge

Strong applied understanding of how AI and ML workloads behave in production, including inference versus training, model and retrieval dependencies, resource utilization, latency, evaluation and monitoring, sensitive‑data handling, and failures of external model services. You should be able to distinguish infrastructure health from model quality, reason about prompt injection and tool permissions with engineering colleagues, and operate AI‑assisted workflows within explicit security and recovery boundaries.

Every team member should be able to discuss the capabilities and limitations of AI relevant to their work, verify AI‑generated output, protect sensitive information, and keep their knowledge current as the technology evolves.

All applicants must be legally authorized to work in the United States without employer sponsorship. Metrosion does not provide employment visa sponsorship.

All positions require working on‑site in our Burlingame, California office five days a week. These roles are not remote or hybrid.

Different perspectives. Shared purpose.

We welcome applicants from all backgrounds and value an inclusive, respectful workplace.