Senior Principal Engineer (AI Engineering)

DSTA; Singapore; On-site; SGD 250,000 - 350,000; Full time

Job summary

DSTA's AI Centre of Excellence is seeking a Senior Principal Engineer to set the technical direction for AI Engineering in Singapore. You will architect large-scale inference systems, define standards and grow deep expertise across the team.

Lead design reviews, build-vs-buy decisions, and technology adoption while mentoring engineers and collaborating with industry partners to strengthen the AI capabilities powering defence operations.

Qualifications

• Degree in Computer Science, Computer Engineering or a related technical field.
• At least 12 years of relevant experience, including 5+ years in technical leadership.
• Deep hands-on expertise in LLM inference and serving in production.
• Strong experience running production workloads on Kubernetes.
• Excellent communication skills to explain complex trade-offs.

Responsibilities

• Define the technical roadmap and reference architectures for AI Engineering.
• Design high-performance, multi-node LLM inference and serving architectures.
• Establish benchmarking, profiling and load-testing practices for model quality and serving performance.
• Architect multi-site AI infrastructure with Kubernetes orchestration and data locality.
• Mentor engineers and drive co-engineering with industry partners.

Skills

LLM inference

Production workloads

Kubernetes

Education

Degree in Computer Science / Computer Engineering

Tools

vLLM

SGLang

TensorRT-LLM

Triton Inference Server

Nsight Systems

PyTorch Profiler

Prometheus

Job description

DSTA's AI Centre of Excellence builds and operates the AI compute platforms, model lifecycle and secure MLOps capabilities that power AI development for Singapore's defence. As we expand our AI footprint, we are looking for a Senior Principal Engineer to set the technical direction for AI Engineering. You will architect large-scale inference and AI systems, establish engineering standards, and grow deep expertise across the team.

Join Us If You

• Want your architecture decisions to shape AI capabilities used in real-world defence operations.
• Enjoy working at every level of the stack, from GPU interconnect and storage to inference runtimes and model governance.
• Are motivated by building a world-class engineering team and culture, not just delivering a product.

Key Responsibilities

What you will be working on

• Technical Leadership. Define the technical roadmap and reference architectures for AI Engineering. Serve primary technical authority on design reviews, build-vs-buy decisions and technology adoption.
• Inference Architecture. Design high-performance, multi-node LLM inference and serving, covering parallelism strategies (tensor, pipeline, expert), disaggregated prefill/decode, KV cache management, batching, speculative decoding and quantisation trade-offs.
• Performance and Cost Engineering. Establish benchmarking, profiling and load-testing practices for model quality and serving performance (e.g. TTFT, TPOT, throughput per GPU). Conduct cost-per-token analysis and right-size accelerator choices to model size, traffic patterns and operational constraints.
• AI Systems Engineering. Architect multi-site AI infrastructure, including multi-cluster Kubernetes orchestration, GPU-aware scheduling, federated observability, cross-site storage and data locality, and model artefact distribution.
• Accelerator and Infrastructure Evaluation. Develop a formal framework to assess GPUs and alternative AI accelerators, along with advanced storage, accelerated interconnect and AI cache/memory tiering.
• Security and Governance. Ensure models deployed for downstream systems are secure, robust and policy-compliant. Lead the automation of MLSecOps controls, including agent-driven scanning, policy gating and remediation, so teams can adopt them with minimal overhead.
• Capability Building. Mentor engineers, shape the team's competency and training roadmap, embed within programme teams to solve problems on the ground, and lead co-engineering effort with industry partners..

Minimum Requirements

• Degree in Computer Science, Computer Engineering or a related technical field.
• At least 12 years of relevant experience, including at least 5 years in a technical leadership capacity.
• Deep hands-on expertise in LLM inference and serving in production.
• Strong experience running production workloads on Kubernetes.
• Excellent communication skills, with the ability to explain complex technical trade-offs to engineers, management and stakeholders.

Preferred Technical Skills

• Serving frameworks and runtimes (e.g. vLLM, SGLang, TensorRT-LLM, Triton Inference Server)
• Model optimisation: quantisation, pruning, distillation, LoRA serving.
• GPU architecture and accelerated interconnect (NVLink/NVSwitch, InfiniBand, RDMA, GPUDirect), and exposure to alternative AI accelerators.
• Parallel file systems, tiered storage and fast model loading for AI workloads.
• Multi-cluster orchestration and service mesh (e.g. Karmada, Istio), autoscaling (e.g. KEDA) and federated observability (e.g. Prometheus federation, distributed tracing).
• Performance tooling (e.g. Nsight Systems/Compute, PyTorch Profiler) and LLM evaluation benchmarks (E.g. GuideLLM).
• MLSecOps, and AI governance frameworks (e.g. AI Verify, ISO/IEC 42001).
• Deploying and operating AI platforms in air-gapped or high-security environments.
• Contributions to open-source AI projects.