Role summary:
We’re looking for a MLOps Engineer to own the infrastructure that powers our AI/ML platform. Our products rely on generative AI models — image generation, video synthesis, LoRA fine-tuning, and LLM orchestration — running across GPU clusters, distributed training pipelines, and multi-environment deployments. We need someone who can make all of that faster, more reliable, and easier to scale.
You’ll be responsible for the full ML infrastructure lifecycle: building and maintaining training pipelines, managing GPU compute clusters, automating model deployment, optimizing inference performance, and ensuring our systems run reliably across cloud, and air-gapped environments. You’ll work closely with our ML engineers and researchers to translate their work into production grade systems.
This is a high-autonomy role on a small team where infrastructure decisions have immediate, visible impact on the product. If you want to build the backbone of an AI company — not just maintain someone else’s — this is the opportunity.
What you will do:
- Design, build, and maintain GPU inference infrastructure on AWS and Azure - including EC2 accelerated computing instances, and SageMaker inference endpoints
- Build and optimize CI/CD pipelines for ML models, from training through evaluation, packaging, and deployment across multiple environments (cloud, air-gapped)
- Automate model deployment workflows — packaging trained models, building optimized inference containers, managing model versioning, and rolling deployments with zero downtime
- Build and maintain custom AMIs and Docker images for GPU workloads using Packer, ensuring consistent environments across development, staging, and production
- Optimize inference performance — GPU memory management, batching strategies and latency reduction for real-time generation endpoints
- Implement monitoring, alerting, and observability for ML systems — GPU utilization, queue depths, model latency, training job health, and cost tracking
- Manage infrastructure as code using Terraform/OpenTofu
- Use AI coding harnesses such as Codex, Claude Code, Pi or Opencode to augment your strong judgement
- Collaborate with ML engineers to debug training failures, optimize data pipelines, and reduce the time from research prototype to production deployment
Work environment:
This role is hybrid but requires occasional in-person work in Reston, VA.
Schedule: Monday – Friday. Occasional off-hours availability for infrastructure incidents, deployment windows, or training job monitoring.
Requirements
Required qualifications:
- U.S. citizenship or permanent residency (required for work with federal customers)
- TS/SCI clearance with polygraph
- 5+ years of experience in infrastructure engineering, DevOps, or MLOps, with at least 2 years specifically supporting ML/AI workloads in production
- Deep hands-on experience with AWS
- Strong proficiency with infrastructure as code (Terraform required; CloudFormation, or CDK also valued)
- Experience building and managing Docker containers and container orchestration for ML workloads
- Production experience with GPU compute management — CUDA environments, NVIDIA driver management, multi-GPU configuration, and GPU memory optimization
- Strong scripting and automation skills in Python and Bash
- Experience with CI/CD systems (GitHub Actions, GitLab CI, or equivalent) including automated testing, build, and deployment pipelines
- Solid understanding of ML model lifecycle: training, evaluation, packaging, versioning, deployment, and monitoring
- Bachelor’s degree in Computer Science, Systems Engineering, or a related technical field (or equivalent professional experience)
Preferred qualifications:
- Familiarity with generative AI model serving – specifically diffusion or GAN-based models
- Background in air-gapped or disconnected environment deployments (NIST 800-171, CUI compliance, classified networks)
- Familiarity with ML experiment tracking tools (Weights & Biases, MLflow, or similar)
- Experience with cost optimization for GPU compute — spot instances, reserved capacity, right-sizing, and usage monitoring
- Prior work supporting federal law enforcement, intelligence community, or defense infrastructure
- Familiarity with Electron apps, or other desktop apps