Senior HPC DevOPS Engineer | TS/SCI w/MD poly required

Power3 Solutions; Hanover (MD); On-site; USD 222,000 - 257,000; Full time

Benefits offered by this job

4 weeks paid time off

11 paid holidays

401k plan with contributions

Annual bonuses

Medical/Dental/Vision insurance

Education and training budget

Certification budget

Job summary

Power3 Solutions seeks a Senior HPC DevOps Engineer to own operations for an HPC/AI compute cluster in Maryland. The role focuses on automation, drift detection, and secure provisioning using Ansible and enterprise tools.

Requirements include 12+ years in CS/IT and a TS/SCI with MD polygraph, with strong Linux, Ansible, and HPC experience. Competitive salary and comprehensive benefits are provided in a demanding, mission-focused environment.

Qualifications

• 12+ years of experience in computer science/IT or related field; BS/MS/PhD pathway specified.
• 7+ years in Linux systems/SRE/DevOps including HPC/large-scale compute environments.
• 3+ years building and operating Ansible automation at scale (roles/collections, idempotency).
• Strong Linux hardening and compliance fundamentals (SELinux/AppArmor, SSH key automation).
• Experience with clustered compute environments and GPU node health validation.

Responsibilities

• Enforce desired state across cluster services via code-driven configuration.
• Automate compute node onboarding and OS provisioning with security baselines.
• Implement rolling maintenance and patch automation; integrate image scanning.
• Ensure auditable logs and integrate with metrics/alerting for incident response.
• Automate common incident responses using runbooks and out-of-band interfaces.
• Maintain docs-as-code for runbooks and operator guidance.

Skills

Linux

Ansible

SRE/DevOps

Git workflows

Automation

Documentation

Education

BS in Computer Science/IT or related field

MS in Computer Science/IT or related field

PhD with 8+ years experience

Tools

Ansible

Container tooling

CI/CD pipelines

Automation controller

Job description

Senior HPC DevOPS Engineer | TS/SCI w/MD poly required

• Hanover, MD

LCAT: IE3

Salary Range: $222- 257K

Location: College Park, MD

Security Clearance Required: TS/SCI with MD polygraph

Job Description:

We are seeking a fully cleared Senior HPC DevOps Engineer to own the operations and automation lifecycle for an existing HPC/AI compute cluster (Linux). You will work closely with team members, as well as directly with our Maryland-based customer, in a fast-paced environment. In this role you will codify repeatable operations in Ansible and drive execution through an enterprise automation controller to enforce desired state, detect drift, accelerate node onboarding, and streamline incident response via runbook automation integrated with monitoring and ITSM.

Key Responsibilities

Desired-state and drift detection: Enforce desired state across cluster services via code-driven configuration; implement drift detection and alert on deviations; reconcile runtime state vs configured state.

Compute node onboarding (Bare-metal/VM): Build and maintain an automated node bootstrap workflow that installs/configures the OS, applies security and performance baselines, enrolls nodes into the scheduler and shared storage ecosystem, validates hardware and service readiness (CPU, network, accelerator, storage mounts), and reports pass/fail results.

Patch & vulnerability response: Implement rolling maintenance and patch automation to meet defined vulnerability response SLAs. Maintain version-controlled container build definitions and integrate image scanning into the build/release lifecycle.

Logging & observability: Ensure automation and operational workflows emit auditable logs to centralized analytics and integrate with metrics/alerting to enable reliable incident response, proactive detection, and safe auto-remediation.

Incident/problem management: Automate responses to common incidents (hung nodes, storage performance alarms, image vulnerabilities, hardware failures) leveraging out-of-band hardware management interfaces and standardized runbooks.

Docs-as-code: Keep runbooks and operational documentation versioned alongside automation and publish operator guidance to the orgs documentation platform.

Required qualifications

12+ years of experience and a BS in computer science, IT, or related technical field, MS and 10 years of experience, or a Ph.D. with 8 years of experience. Four years of additional experience is required in lieu of a Bachelors’ degree for a total of 16 years of experience.

7+ years in Linux systems / SRE / DevOps, including production cluster operations in an HPC or large-scale compute environment.

3+ years of experience building and operating Ansible automation at scale (roles/collections, idempotency, inventories, secrets).

Strong Linux hardening & compliance fundamentals (SELinux/AppArmor, SSH key automation, baseline config management).

Demonstrated experience operating or automating clustered compute environments (HPC, large Linux farms, or similar).

Hands‑on experience with container tooling in Linux environments, including image lifecycle/versioning.

Familiarity with incident response and runbook-driven operations; ability to automate common remediations.

Strong Git workflow and documentation practices.

Must hold at least one active/current technical certification from the following-

• Information security (e.g., CISSP)
• Networking (e.g., CCNA)
• System Administration (e.g., RHCE, MCSE)
• IT systems management (e.g., ITIL)

This position requires an active/current TS/SCI w/ Polygraph.

Preferred qualifications

• Bare-metal provisioning experience (PXE/iPXE, Kickstart/Preseed, Foreman/MAAS) and hardware OOB management.
• CI/testing for automation and promotion pipelines for playbooks
• Experience with tuned performance profiles, HPC performance troubleshooting, and GPU node health validation.
• Experience generation operational documentation from repos (Confluence).

Benefits

• Four (4) weeks paid time
• Eleven Paid Holidays
• 401k plan with 4% employer contributions and immediate vesting plus annual 3% contruibution
• Annual Bonuses for performance
• Medical, Dental, and Vision and insurance
• Education and Training budget provided
• Technical Certification and coursework budget for computer/program expenses