Senior LLM Performance & Evaluation Engineer

Bitdeer Group; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Bitdeer is seeking an experienced ML infrastructure engineer to build and maintain benchmark pipelines, define model launch gates, and simulate realistic workloads at scale.

You will evaluate model and runtime options, implement regression detection in CI/CD, and collaborate with SRE to turn benchmarks into actionable SLOs and alerts. The role emphasizes performance, cost, and reliability in a fast‑growing AI and mining tech environment.

Qualifications

• 5+ years in ML infrastructure, performance engineering, model evaluation, QA automation, or backend testing for production systems.
• Experience with LLM serving metrics such as TTFT, TPOT/ITL, request latency, token throughput, concurrency, and GPU utilization.
• Strong Python skills for benchmark/evaluation automation; Go experience is a plus for integrating with platform services.
• Familiarity with OpenAI/Anthropic APIs, vLLM/Dynamo/SGLang/Triton‑style servers, and Kubernetes‑based test environments.
• Able to design statistically meaningful tests and communicate tradeoffs between model quality, latency, reliability, cost, and customer experience.
• Experience building dashboards, reports, and release gates for engineering, product, and business stakeholders.

Responsibilities

• Build benchmark pipelines for TTFT, inter‑token latency, output throughput, request latency, concurrency curves, error rate, and GPU utilization.
• Create model launch gates covering OpenAI/Anthropic compatibility, streaming behavior, tool calling, reasoning output, multimodal behavior, and long‑context cases.
• Maintain representative workloads using synthetic, replayed, and customer‑like traffic patterns for both steady‑state and burst scenarios.
• Compare model/runtime/provider options and publish clear recommendations for routing, fallback, pricing, and capacity decisions.
• Automate regression detection in CI/CD and staging so model, runtime, or configuration changes do not silently degrade quality, latency, or cost.
• Work with runtime engineers to identify bottlenecks and verify improvements; work with SRE to convert benchmark results into SLOs and alert thresholds.

Skills

Python

OpenAI API

Anthropic API

Benchmarking

Model evaluation

QA automation

Performance engineering

Go

Kubernetes

vLLM

Dynamo

SGLang

Triton

Tools

vLLM

Dynamo

SGLang

Triton

Job description

Bitdeer is seeking an experienced ML infrastructure engineer to build and maintain benchmark pipelines, define model launch gates, and simulate realistic workloads at scale.

You will evaluate model and runtime options, implement regression detection in CI/CD, and collaborate with SRE to turn benchmarks into actionable SLOs and alerts. The role emphasizes performance, cost, and reliability in a fast‑growing AI and mining tech environment.