Senior MLOps Engineer: LLM Production & SRE

Home Team Science and Technology Agency (HTX); Singapore; On-site; SGD 120,000 - 170,000; Full time

Job summary

HTX, Singapore's science and technology agency for homeland security, seeks an experienced MLOps/SRE Engineer to deploy and operate a production LLM system within a secure infrastructure. You will own end-to-end responsibilities from model inference to vector databases, observability, and cost control.

The role requires 4+ years in MLOps/SRE/DevOps, hands-on production LLM experience, strong Kubernetes skills, and a proactive mindset for reliability and security.

Qualifications

• 4+ years in MLOps, SRE, or DevOps roles.
• Hands-on with deploying and operating LLMs in production.
• Strong Kubernetes expertise including operators, StatefulSets, and GPU scheduling.

Responsibilities

• Deploy and manage LLM models on GPU infrastructure using vLLM/TensorRT-LLM.
• Provision and maintain infrastructure including vector databases and orchestration services.
• Profile and optimize LLM inference performance and batching strategies.
• Implement monitoring with Prometheus, Grafana, DCGM exporters, and Elastic Stack.
• Establish SLOs/SLIs, auto-scaling policies, and incident recovery mechanisms.
• Cost optimization to reduce token usage and compute spend.
• Ensure security, manage secrets, and maintain audit logs in regulated environments.
• Participate in on-call rotation and incident analysis.
• Capacity planning to scale platform with adoption.

Skills

MLOps

SRE

DevOps

Kubernetes

GPU scheduling

Python

Bash

Terraform

Helm

Kustomize

Milvus

Weaviate

Qdrant

Pinecone

Kubeflow

MLflow

Prometheus

Grafana

Tools

Terraform

Helm

Kustomize

Kubeflow

MLflow

Milvus

Weaviate

Qdrant

Pinecone

Docker

DCGM

Kong Gateway

Job description

HTX, Singapore's science and technology agency for homeland security, seeks an experienced MLOps/SRE Engineer to deploy and operate a production LLM system within a secure infrastructure. You will own end-to-end responsibilities from model inference to vector databases, observability, and cost control.

The role requires 4+ years in MLOps/SRE/DevOps, hands-on production LLM experience, strong Kubernetes skills, and a proactive mindset for reliability and security.