Senior Site Reliability Engineer AI Security Federal

Bizfirst; McLean (VA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Medical, dental, and vision coverage

401(k) employer matching

Tuition reimbursement

Job summary

BizFirst is recruiting a Senior Site Reliability Engineer for AI security within a federal civilian agency environment. You will keep a self-managed, distributed AI runtime security platform on AWS EKS healthy, fast, and compliant, partnering with DevOps and security teams to improve reliability over time.

The role involves operating production Kubernetes clusters, implementing IAM, network controls, and continuous monitoring, while planning upgrades and embracing Terraform/Helm for IaC. U.S.

Qualifications

• Five years of experience in site reliability, DevOps, or cloud infrastructure engineering.
• Experience operating production Kubernetes clusters with monitoring, scaling, and troubleshooting.
• Experience with core AWS services (IAM, VPC, EC2, load balancing, KMS, Secrets Manager, CloudWatch).
• Experience managing infrastructure as code with Terraform.
• Scripting or programming experience in Python, Go, or Bash.
• Ability to obtain and maintain a DHS Public Trust determination.

Responsibilities

• Operate, monitor, and troubleshoot a distributed, high-throughput platform on Amazon EKS.
• Plan and execute upgrades to Kubernetes versions and add-ons with minimal downtime.
• Manage infrastructure as code with Terraform and Helm.
• Design autoscaling, capacity, and load testing to meet latency targets.
• Build and maintain metrics, logs, traces, and alerting; define SLOs; lead incident reviews.
• Implement security requirements, including Secrets Manager, IAM, network controls, and image hardening.

Skills

Site reliability

DevOps

Cloud infrastructure

Education

Bachelor's degree in Computer Science or related field

Tools

Terraform

Kubernetes

AWS

Python

Go

Bash

Job description

Senior Site Reliability Engineer AI Security Federal

About the Role

BizFirst is assisting our client with recruiting a Senior Site Reliability Engineer for AI security in a federal civilian agency environment. As the resident reliability engineer, you will keep a self-managed, distributed AI runtime security platform on AWS EKS healthy, fast, and compliant. The platform handles the agency's AI requests, making its availability and latency important to mission systems.

You will work alongside the agency's DevOps and security teams to operate, scale, and upgrade the platform, and partner with our client's engineering team to improve reliability over time.

Ourclient is a technology integrator that supports government customers with ITmodernization, performance engineering, and cybersecurity. Its engineeringteams collaborate on complex technical programs in a culture of innovation,continuous improvement, and professional development.

What You Will Do

Operate, monitor, and troubleshoot a distributed, high-throughput platform on Amazon EKS, addressing pod and node issues, networking, storage, and performance bottlenecks.

Plan and execute upgrades to the platform, Kubernetes versions, and add-ons with minimal downtime and within agency change management.

Manage infrastructure as code with Terraform and Helm, keeping environments consistent from development through production.

Design and tune autoscaling, capacity, and load testing to meet latency and throughput targets as demand grows.

Build and maintain metrics, logs, traces, and alerting; define service level objectives (SLOs); and lead incident response and post-incident reviews.

Implement customer security requirements, including secrets management with AWS Secrets Manager and KMS, least-privilege IAM, network controls, image hardening, patching, and evidence for authorization to operate (ATO) and continuous monitoring.

Required Qualifications

Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Five years of experience in site reliability, DevOps, or cloud infrastructure engineering.

Experience operating production Kubernetes clusters, including monitoring, scaling, and troubleshooting containerized workloads.

Experience with core AWS services, including IAM, VPC, EC2, load balancing, KMS, Secrets Manager, and CloudWatch.

Experience managing infrastructure as code with Terraform.

Experience scripting or programming in Python, Go, or Bash.

Ability to obtain and maintain a DHS Public Trust determination.

Helpful Experience

The following qualifications are preferred, not required:

Experience running Amazon EKS, including managed node groups, Karpenter or Cluster Autoscaler, IRSA or EKS Pod Identity, and the AWS Load Balancer Controller.

Experience operating latency-sensitive, high-throughput services, including load testing and performance tuning.

Experience with Prometheus, Grafana, OpenTelemetry, and forwarding logs to a SIEM such as Splunk.

Experience with Helm and GitOps or CI/CD for Kubernetes, including Argo CD, Flux, GitHub Actions, or GitLab CI.

Experience in federal environments, including ATO, NIST SP 800-53, AWS GovCloud, FIPS endpoints, and STIG-hardened container images.

Experience with GPU node pools or serving machine learning models on Kubernetes.

Familiarity with AI and LLM applications and traffic patterns, including streaming responses and large payloads.

Certifications such as CKA, CKS, AWS Solutions Architect or SysOps Administrator, or HashiCorp Terraform Associate.

Benefits Provided by Our Client

Medical, dental, and vision coverage.

Employer matching contributions to a 401(k) retirement plan.

Competitive compensation and bonus opportunities.

Support for continuing education and professional growth, including tuition reimbursement.

Paid leave and holidays.

Our client is an Equal Opportunity Employer and a U.S. Federal Government contractor. All qualified applicants receive consideration without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, veteran status, or any other protected characteristic under applicable federal, state, or local laws.

Our client provides reasonable accommodations to individuals with disabilities throughout the application and employment process. Applicants who need assistance or an accommodation due to a disability may contact our client.

U.S. citizenship is required for all positions with a government clearance and certain other restricted positions.

Employment: Full-time

Public Trust: Ability to obtain and maintain a DHS Public Trust determination

BizFirst LLC is an Equal Opportunity Employer and does not discriminate on the basis of race or ethnicity, religion, sex, national origin, age, veteran status, disability, or genetic information, or any other reason prohibited by law in employment.