Lead MLOps & SRE Engineer — Developer Experience

HTX (Home Team Science & Technology Agency); Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

HTX (Home Team Science & Technology Agency) is seeking a seasoned MLOps/SRE/DevOps specialist to deploy and operate production-grade LLMs on GPU infrastructure. You will manage vector databases, orchestration services, and API gateways, while tuning performance and ensuring reliable, observable systems.

Applicants should bring 4+ years in ML Ops and strong Kubernetes expertise, plus IaC experience and familiarity with vector DBs, Ray/Kubeflow/MLflow. Singapore-based role with a two-year contract.

Qualifications

• 4+ years of experience in MLOps, SRE, or DevOps roles, with at least 1 year working with ML/AI systems.
• Hands-on experience deploying and operating LLMs in production (vLLM, TGI, TensorRT-LLM, or similar).
• Strong Kubernetes expertise including operators, StatefulSets, and GPU scheduling.
• Deep understanding of GPU architecture, and inference optimization techniques.
• Experience with observability tools (Prometheus, Grafana, ELK/Elastic Stack).
• Solid Python and Bash scripting skills for automation.
• Knowledge of vector databases (Milvus, Weaviate, Qdrant, or Pinecone).
• Experience with infrastructure-as-code (Terraform, Helm, Kustomize).
• Experience with NVIDIA GPUs (A100/H100/B200) and DCGM monitoring.
• Understanding of LLM inference concepts: KV cache, continuous batching, PagedAttention.
• Familiarity with Ray clusters, Kubeflow, or MLflow.
• Background in SRE practices: SLO/SLI definition, error budgets, incident management.
• Experience with secure or regulated environments.
• Knowledge of LiteLLM, Kong Gateway, or API management platforms.
• Systems thinking with ability to diagnose complex issues across the ML stack.
• Data-driven decision making using metrics and telemetry.
• Proactive mindset focused on reliability, automation, and preventive measures.
• Strong debugging skills for GPU, networking, and distributed systems issues.
• Clear incident communication and documentation.
• Collaborative approach working with data scientists, ML engineers, and platform teams.

Responsibilities

• LLM Deployment: Deploy and manage LLM models using vLLM/TensorRT-LLM on our GPU infrastructure, optimizing for throughput, latency, and GPU utilization.
• Infrastructure Management: Provision and maintain supporting infrastructure including vector databases, orchestration services, Redis/queue systems, and API gateways.
• Performance Optimization: Profile and tune LLM inference performance, experiment with batching, context caching, and quantization techniques.
• Observability: Implement monitoring using Prometheus, Grafana, DCGM exporters, and Elastic Stack to track latency, throughput, cache hits, and health.
• Reliability Engineering: Establish SLOs/SLIs, implement auto-scaling, design failure recovery, and conduct chaos engineering for uptime.
• Cost Optimization: Monitor GPU utilization and inference costs, identify opportunities, reduce token usage and compute spend.
• Security & Compliance: Ensure components operate within secure boundaries, manage secrets, and maintain audit logs.
• Incident Response: Participate in on-call rotation, troubleshoot incidents, perform root cause analysis, implement preventive measures.
• Capacity Planning: Model future load, forecast GPU requirements, and coordinate with infra teams to scale.

Skills

Kubernetes

Python

Bash

GPU-architecture

Observability

SRE-practices

Terraform

Helm

Kustomize

DCGM

Vector-databases

Ray Kubeflow MLflow

Tools

Milvus

Weaviate

Qdrant

Pinecone

vLLM

TensorRT-LLM

TGI

Kubeflow

MLflow

NVIDIA GPUs

DCGM monitoring

Job description

HTX (Home Team Science & Technology Agency) is seeking a seasoned MLOps/SRE/DevOps specialist to deploy and operate production-grade LLMs on GPU infrastructure. You will manage vector databases, orchestration services, and API gateways, while tuning performance and ensuring reliable, observable systems.

Applicants should bring 4+ years in ML Ops and strong Kubernetes expertise, plus IaC experience and familiarity with vector DBs, Ray/Kubeflow/MLflow. Singapore-based role with a two-year contract.