Lead HPC Systems Engineer

EPAM Systems Inc; Newtown (PA); Remote; USD 155,000 - 175,000; Full time

Benefits offered by this job

Medical Insurance

Health Savings Account

Flexible Spending Accounts

Disability Insurance

Life Insurance

Employee Assistance Program

LinkedIn Learning

401(k) Matching

Paid Time Off

Paid Holidays

Legal Plan

Identity Theft Protection

Pet Insurance

Employee Stock Purchase Program

Performance Bonus

Job summary

EPAM Systems, Inc. is seeking an experienced engineer to own the reliability of a multi-cloud HPC AI infrastructure. You will investigate hard operational problems across compute, accelerators, storage, and networking, and translate findings into automated remediation with safe AI-powered agent workflows.

Success comes from deep production experience with HPC, GPUs, and distributed systems, plus the ability to strengthen observability and drive durable fixes across teams and clouds.

Qualifications

• Production HPC or large-scale distributed compute experience in reliability-focused engineering.
• Strong Linux and networking debugging across containers, storage I/O, DNS and authentication.
• Ability to correlate telemetry across cloud environments using multiple tools rather than a single dashboard.
• Automation coding in Python or Go and ownership of complex production investigations to final fixes.

Responsibilities

• Understand multi-cloud HPC architecture and keep it current as infrastructure evolves.
• Proactively identify reliability and performance issues by correlating logs, metrics and events.
• Diagnose end-to-end across Linux hosts, containers, schedulers, and accelerators to root cause.
• Develop repeatable tooling and agent workflows for safe detection, diagnosis and remediation.
• Improve observability and monitor automation effectiveness with cross-team collaboration.

Skills

HPC operations

Linux networking

Telemetry correlation

Automation in Python/Go

Tools

Prometheus

Grafana

OpenTelemetry

Centralized logging

Job description

Our client is looking for an engineer with high-performance computing (HPC) and site-reliability experience to keep their growing multi-cloud AI infrastructure reliable and efficient. You will investigate hard operational problems across compute, accelerators, workload scheduling, storage, and networking, and turn what you learn into proactive detection, diagnosis, and safe automated remediation, including AI-powered agents.

Success here starts with understanding how HPC infrastructure behaves in production and learning unfamiliar systems fast. Prior experience building or orchestrating AI agents is welcome but not required.

Req# 1103823066

Responsibilities

• Build a working understanding of multi-cloud HPC architecture (clusters, schedulers, accelerators, storage, networking, authentication) and keep it current as the infrastructure changes

• Find reliability and performance problems proactively, correlating logs, metrics, and events across cloud providers to build a clear incident picture and spot gaps in telemetry

• Diagnose complex issues end to end across Linux hosts, containers, schedulers, distributed workloads, storage, networking, and accelerators, isolating root cause rather than treating symptoms

• Turn proven investigations into repeatable tooling and agent-enabled workflows for detection, diagnosis, and bounded remediation, with safety built in: scoped permissions, precondition checks, human approval where needed, rollback, and post-change verification

• Strengthen observability and monitoring, measure whether the automation actually works (accuracy, recovery time, recurrence, false positives), and partner with infrastructure, research, security, and cloud-provider teams to turn incident lessons into durable fixes

Requirements

• Operations experience with production HPC, GPU/accelerator clusters, or large-scale distributed compute in a reliability-focused engineering role

• Strong Linux and networking skills, debugging failures across processes, containers, storage I/O, DNS, connectivity, and authentication

• Correlates telemetry across multiple cloud environments using tools such as Prometheus, Grafana, OpenTelemetry, or centralized logging, rather than relying on a single dashboard

• Writes and maintains automation in a language such as Python or Go, and has owned complex production investigations from symptom through root cause to a verified, lasting fix

Nice to have

• Hands-on with HPC scheduling or cluster orchestration (Slurm, Kubernetes, or similar), and an understanding of how infrastructure health affects workload placement and completion

• Supporting AI training or accelerator-intensive workloads: GPU/TPU health, distributed-job failures, interconnect problems, data-loading bottlenecks, or checkpointing

• Experience across clouds such as AWS, GCP, CoreWeave, or OCI and their native observability services. You do not need every provider named here

• HPC storage and networking (NFS/POSIX, Lustre, object storage, RDMA/InfiniBand or RoCE), infrastructure-as-code and deployment tooling (Terraform, Helm, cluster operators), and failure-injection or replayable incident testing

• Building, integrating, evaluating, or orchestrating AI agents for infrastructure operations (tool interfaces, permissions, verifying agent actions)

• Welcome, but not required; strong HPC operations and SRE skills are the foundation of this role

We offer

• Medical, Dental and Vision Insurance (Subsidized)

• Health Savings Account

• Flexible Spending Accounts (Healthcare, Dependent Care, Commuter)

• Short-Term and Long-Term Disability (Company Provided)

• Life and AD&D Insurance (Company Provided)

• Employee Assistance Program

• Unlimited access to LinkedIn learning solutions

• Matched 401(k) Retirement Savings Plan

• Paid Time Off – the employee will be eligible to accrue 15-25 paid days, depending on specific level and tenure with EPAM (accrual eligibility may change over time)

• Paid Holidays - nine (9) total per year

• Legal Plan and Identity Theft Protection

• Accident Insurance

• Employee Discounts

• Pet Insurance

• Employee Stock Purchase Program

• If otherwise eligible, participation in the discretionary annual bonus program

• If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program

This Remote Position Cannot be Performed in New York City.

This posting includes a good faith range of the salary EPAM would reasonably expect to pay the selected candidate. The range provided reflects base salary only. Individual compensation offers within the range are based on a variety of factors, including, but not limited to: geographic location, experience, credentials, education, training; the demand for the role; and overall business and labor market considerations. Most candidates are hired at a salary within the range disclosed. Salary range: $155,000 - $175,000. In addition, the details highlighted in this job posting above are a general description of all other expected benefits and compensation for the position.

In accordance with the LA County Fair Chance Ordinance, you may find a copy of the Notice containing a summary of the Ordinance’s key provisions here: Concept FCO Posting 8 27 24 (lacounty.gov)

EPAM Systems, Inc. is an equal opportunity employer. We recognize the value of diversity and inclusion in creating success for our customers, business partners, shareholders, employees and communities. We are committed to recruiting, hiring, developing and promoting employees without discrimination. As a global employer, this commitment includes complying with all laws in the countries in which we operate. Nevertheless, we believe equal employment practices should not be limited to what the law requires. Equal opportunity and inclusion are essential to motivate, empower and recognize the best in everyone.

At EPAM, employment actions are based on individual qualifications, without regard to race, color, religion, creed, gender, pregnancy status, sexual orientation, gender identity, gender expression, marital or familial status, national origin, ancestry, genetics, age, disability status, veteran status, citizenship status when otherwise legally able to work, or any other characteristic protected by law.