ML Engineer (LLM Systems)

Cynnovative; Arlington (VA); On-site; USD 120,000 - 150,000; Full time

Job summary

Cynnovative is seeking a Senior ML Engineer in Arlington, Virginia to develop and manage tools for LLM experimentation and deployment. You will design scalable LLM systems and ensure the reliability of these systems in production environments. The ideal candidate must have strong expertise in ML frameworks like PyTorch, containerization, and cloud platforms, alongside a B.S. in a related field. A TS/SCI security clearance is mandatory. This role plays a pivotal role in supporting U.S. national security efforts.

Qualifications

• U.S. Citizenship and active TS/SCI security clearance required.
• Experience with high-performance LLM inference frameworks is a plus.
• Understanding of ML system tradeoffs (latency vs throughput vs cost) is beneficial.

Responsibilities

• Design and build scalable LLM systems for high-throughput experimentation and inference.
• Lead development of experimentation infrastructure.
• Ensure production readiness and operational reliability of LLM systems.
• Collaborate cross-functionally with applied mathematicians and research engineers.

Skills

Strong communication skills and cross-functional collaboration

Deep understanding of transformer architectures and LLM inference workflows

Hands-on experience building scalable ML systems

Proficiency in Python and ML frameworks (e.g., PyTorch, Hugging Face)

Experience with distributed systems or large-scale compute environments

Experience with containerization and cloud platforms (Docker, Kubernetes, AWS/GCP/Azure)

Familiarity with CI/CD workflows for ML systems

Experience with version control systems (Git)

Education

B.S. in Computer Science, Software Engineering, or related field (M.S. or Ph.D. preferred)

Tools

Docker

Kubernetes

AWS

GCP

Azure

PyTorch

Hugging Face

vLLM

SGLang

Job description

At Cynnovative, we leverage machine learning, computer science, and software engineering to address high-impact problems in the cyber domain, specifically those which are critical to U.S. national security. We primarily extend fundamental research to invent, design, develop, and deploy prototype solutions that support persistent problems in this domain.

Job Overview

As a Senior ML Engineer (LLM Systems) at Cynnovative, you will be responsible for developing and managing tools that facilitate LLM experimentation and deployment. This role is crucial in ensuring seamless integration and operation of machine learning models in various environments, supporting U.S. national security efforts.

NOTE: This role requires an active TS/SCI security clearance and is located on-site in Northern Virginia.

Responsibilities May Include

Design and build scalable LLM systems for high-throughput experimentation and inference

• Optimize inference performance (latency, throughput)
  • Batching, caching, and request scheduling
  • Efficient GPU/CPU utilization and memory management
• Design and deploy containerized ML services (e.g., Docker, Kubernetes)

Lead development of experimentation infrastructure

• Build frameworks for large-scale experiment sweeps and parallel execution
• Ensure fault tolerance, retry logic, and reproducibility

Ensure production readiness and operational reliability of LLM systems

• Implement testing strategies and validation pipelines
• Design APIs and model serving systems
• Maintain observability (logging, monitoring, tracing)
• Debug and resolve issues in production systems
• Deploy systems in secure or constrained environments

Collaborate cross-functionally

• Work closely with applied mathematicians and research engineers
• Provide technical leadership and mentorship
• Establish engineering best practices

Requirements Must Have

• B.S. in Computer Science, Software Engineering, or related field (M.S. or Ph.D. preferred)
• Strong communication skills and cross-functional collaboration
• Deep understanding of transformer architectures and LLM inference workflows
• Hands-on experience building scalable ML systems
• Proficiency in Python and ML frameworks (e.g., PyTorch, Hugging Face)
• Experience with distributed systems or large-scale compute environments
• Experience with containerization and cloud platforms (Docker, Kubernetes, AWS/GCP/Azure)
• Familiarity with CI/CD workflows for ML systems
• Experience with version control systems (Git)
• U.S. Citizenship and active TS/SCI security clearance

Desired Skills Nice To Have

• Experience with high-performance LLM inference frameworks (vLLM, SGLang, etc.)
• Understanding of ML system tradeoffs (latency vs throughput vs cost)
• Experience bridging research and production systems
• Familiarity with cyber-related data, tools, and techniques