Senior Kubernetes Engineer

Peraton; Herndon (VA); Remote; USD 140,000 - 180,000; Full time

Job summary

Peraton is seeking an experienced Senior Kubernetes Engineer to support modernization and migration of containerized workloads in a government multi-cloud environment with 70+ tenants. You will design AKS architectures, stand up ARK on Kubernetes, and mentor teams while driving incident RCA and ensuring FedRAMP compliance and security best practices.

The role requires leadership, deep Kubernetes/ARK expertise, and strong collaboration with architecture and security teams.

Qualifications

• Bachelor's degree or higher with 6–12 years of experience depending on degree
• U.S. Citizenship with ability to obtain DHS Public Trust clearance
• Hands-on Kubernetes in production environments, esp. AKS
• Experience deploying ARK or comparable Kubernetes-native agents
• Strong incident management and root cause analysis capabilities
• Proven ability to communicate with technical teams and senior stakeholders

Responsibilities

• Lead Tier 3 incident escalation and resolution for AKS/ARK environments
• Drive root cause analysis and implement preventive solutions
• Define observability standards using Azure Monitor, Prometheus, OpenTelemetry
• Mentor Tier 2 engineers on Kubernetes and ARK operations
• Enforce security, FedRAMP controls, and RBAC policies
• Collaborate with Architecture to align with Azure Well-Architected Framework
• Maintain thorough incident response documentation

Skills

Kubernetes administration

AKS production experience

Incident management

FedRAMP security awareness

leadership & communication

technical documentation

Education

Bachelor's degree in engineering or related field

Tools

ARK

ARK CRDs & Controllers

Terraform

Helm

Argo CD / Flux

Azure Kubernetes Service (AKS)

GitHub Actions

Job description

Basic Qualifications:

• Bachelors degree and 8 years of experience or a Masters degree and 6 years of experience or an Associates degree and 10 year of experience or a High School diploma/equivalent and 12 years of experience.
• Must be a U.S. Citizen with the ability to obtain and maintain a DHS Public Trust clearance.
• Engineering and systems analysis experience, with a deep hands‑on focus on Kubernetes in production
• Extensive hands‑on expertise operating Azure Kubernetes Service (AKS) and the broader Kubernetes stack — control plane, CNI networking, ingress, CSI storage, RBAC, Helm, and workload scheduling
• Hands‑on experience deploying and operating Agentic Runtime for Kubernetes (ARK) or comparable Kubernetes‑native agent or operator platforms built on the CRD‑plus‑controller pattern (e.g., Argo), including ARK Custom Resources (Agents, Models, Teams, Tools, and Queries) and MCP/A2A integration
• Proven experience leading incident management and root cause analysis in large‑scale, multi‑tenant government environments, with deep knowledge of FedRAMP compliance and Kubernetes and Azure security best practices
• Strong leadership, communication, and technical documentation skills; ability to communicate effectively with both technical teams and senior stakeholders

Preferred Qualifications:

• Certified Kubernetes Administrator (CKA) or Certified Kubernetes Application Developer (CKAD), plus Microsoft Azure Administrator Associate (AZ‑104) or Microsoft Azure Solutions Architect Expert (AZ‑305)
• Proficiency with GitOps and Infrastructure as Code (Terraform, Bicep, Helm, and Argo CD or Flux) and CI/CD pipelines (Azure DevOps or GitHub Actions)
• Advanced scripting with Python, PowerShell, or Go for automation and remediation, including use of the ARK Python SDK (ark-sdk)
• Experience integrating LLM providers (Azure OpenAI, Anthropic, Google, or local models) behind an agentic runtime

Peraton is seeking an experienced Senior Kubernetes Engineer to support development, modernization, and migration of containerized and agentic workloads in a government multi‑cloud environment with 70+ customer tenants and growing.

The applicant must have extensive engineering experience and deep Kubernetes and Azure expertise, including designing and leading implementation of container‑platform requirements and architecture on Azure Kubernetes Service (AKS), as well as standing up and operating Agentic Runtime for Kubernetes (ARK) to host AI‑agent workloads. The Senior Kubernetes Engineer will serve as a technical escalation point, drive root cause analysis on complex incidents, and provide mentorship and knowledge sharing across engineering teams. A passion for performance, strong desire for quality conformance, and attention to detail are prerequisites for this position.

Location: Remote

Work Hours: This position requires working online from 8:00 AM Eastern to 5:00 PM Eastern

Day to Day Roles and Responsibilities:

Incident Response and Troubleshooting

• Serve as the Tier 3 escalation point for complex incidents and outages across the Azure Kubernetes Service (AKS) and ARK agentic‑runtime environment
• Lead investigation and resolution of issues across the Kubernetes stack including AKS clusters, the control plane, CNI networking, ingress, CSI storage, and workload scheduling, as well as the ARK control plane and its Custom Resources (Agents, Models, Teams, Tools, and Queries)
• Drive root cause analysis on high‑severity incidents and own implementation of solutions to prevent recurrence
• Manage high‑priority incident communications with stakeholders, leadership, and affected tenants

Platform Monitoring and Optimization

• Define and enforce observability standards across tenant clusters using Azure Monitor, Managed Prometheus, Container Insights, Azure Log Analytics, and OpenTelemetry, including the agent‑level telemetry emitted by ARK
• Lead proactive identification and mitigation of cluster, node‑pool, and workload risks before they impact tenants
• Analyze performance, cost, and capacity trends and architect optimizations to improve stability, efficiency, and security at scale

Kubernetes and ARK Platform Engineering

• Lead management and maintenance of Azure Kubernetes Service (AKS) clusters and the ARK agentic runtime across a 70+ tenant environment
• Deploy, upgrade, and operate ARK on Kubernetes — managing its CRDs, controllers, and model and tool integrations (including MCP servers) to host AI agents declaratively and provider‑agnostically
• Ensure all tenant resources are secure, FedRAMP compliant, and continuously optimized, enforcing Kubernetes RBAC, network policy, and Azure workload identity
• Partner with the Architecture team to design solutions aligned with the Azure Well‑Architected Framework and Kubernetes best practices, and own and enforce Change Management procedures across the team

Collaboration, Mentorship, and Knowledge Sharing

• Serve as technical lead for incident intake, triage prioritization, and team workload management
• Mentor and train Tier 2 engineers on Kubernetes and agentic‑runtime operations; lead knowledge transfer sessions to build team capability
• Drive improvements in team metrics and service delivery in coordination with the Service Delivery Manager and Product Owner
• Maintain comprehensive documentation of incident response procedures, architecture decisions, and lessons learned