Lead HPC Systems Engineer

EPAM Systems Inc; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

EPAM Systems Inc. is seeking an engineer with HPC and site-reliability experience to maintain a growing multi-cloud AI infrastructure. You will troubleshoot complex issues across compute, accelerators, storage, and networking, and turn findings into automated remediation with safety checks.

Prior experience with AI agents is a plus. Responsibilities include building understanding of multi-cloud HPC architecture, proactive problem detection, end-to-end diagnostics, and developing repeatable

Qualifications

• Production HPC experience required.
• Experience with GPU/accelerator clusters or large-scale distributed compute.
• Strong Linux and networking debugging skills.
• Experience writing automation in Python or Go and ownership of incidents.

Responsibilities

• Build understanding of multi-cloud HPC architecture and keep it current as infrastructure evolves.
• Proactively identify reliability and performance problems by correlating logs, metrics, and events across clouds.
• Diagnose complex end-to-end issues across Linux, containers, schedulers, storage, networking, and accelerators.
• Turn investigations into repeatable tooling and agent-enabled workflows with safe automated remediation.
• Improve observability and monitoring and translate incident lessons into durable fixes.

Skills

HPC operations

Linux administration

Networking

Observability

Automation

Python

Go

Incident investigation

Tools

Prometheus

Grafana

OpenTelemetry

Logging systems

Job description

Our client is looking for an engineer with high-performance computing (HPC) and site-reliability experience to keep their growing multi-cloud AI infrastructure reliable and efficient. You will investigate hard operational problems across compute, accelerators, workload scheduling, storage, and networking, and turn what you learn into proactive detection, diagnosis, and safe automated remediation, including AI-powered agents. Success here starts with understanding how HPC infrastructure behaves in production and learning unfamiliar systems fast. Prior experience building or orchestrating AI agents is welcome but not required. Req# 1103823066

Responsibilities

• Build a working understanding of multi-cloud HPC architecture (clusters, schedulers, accelerators, storage, networking, authentication) and keep it current as the infrastructure changes
• Find reliability and performance problems proactively, correlating logs, metrics, and events across cloud providers to build a clear incident picture and spot gaps in telemetry
• Diagnose complex issues end to end across Linux hosts, containers, schedulers, distributed workloads, storage, networking, and accelerators, isolating root cause rather than treating symptoms
• Turn proven investigations into repeatable tooling and agent-enabled workflows for detection, diagnosis, and bounded remediation, with safety built in: scoped permissions, precondition checks, human approval where needed, rollback, and post-change verification
• Strengthen observability and monitoring, measure whether the automation actually works (accuracy, recovery time, recurrence, false positives), and partner with infrastructure, research, security, and cloud-provider teams to turn incident lessons into durable fixes

Requirements

• Operations experience with production HPC, GPU/accelerator clusters, or large-scale distributed compute in a reliability-focused engineering role
• Strong Linux and networking skills, debugging failures across processes, containers, storage I/O, DNS, connectivity, and authentication
• Correlates telemetry across multiple cloud environments using tools such as Prometheus, Grafana, OpenTelemetry, or centralized logging, rather than relying on a single dashboard
• Writes and maintains automation in a language such as Python or Go, and has owned complex production investigations from symptom through root cause to a verified, lasting fix

Nice to have

• Hands-on with HPC scheduling or cluster orchestration (Slurm, Kubernetes, or similar), and an understanding of how infrastructure health affects workload placement and completion
• Supporting AI training or accelerator-intensive workloads: GPU/TPU health, distributed-job failures, interconnect problems, data-loading bottlenecks, or checkpointing
• Experience across clouds such as AWS, GCP, CoreWeave, or OCI and their native observability services
• You do not need every provider named here HPC storage and networking (NFS/POSIX, Lustre, object storage, RDMA/InfiniBand or RoCE), infrastructure-as-code and deployment tooling (Terraform, Helm, cluster operators), and failure-injection or replayable incident testing
• Building, integrating, evaluating, or orchestrating AI agents for infrastructure operations (tool interfaces, permissions, verifying agent actions)
• Welcome, but not required; strong HPC operations and SRE skills are the foundation of this role