AI Testing Lead - LLM Evaluation & Safety

NCS PTE. LTD.; Singapore; On-site; SGD 72,000 - 120,000; Full time

Job summary

NCS PTE. LTD. in Singapore is seeking an AI Tester to lead end-to-end evaluation of AI applications, including LLM features, RAG systems, and multi-agent workflows.

You will own test design, execution, and reporting across offline, pre-prod, and in-prod stages, and collaborate with product, data, and platform teams. The role focuses on non-functional testing, security, observability, and CI/CD integration, with responsibilities spanning test automation, ROI metrics, and clear risk-informed

Qualifications

• 3+ years of software testing/QA with API and performance testing.
• Experience with CI/CD and version control.
• Understanding of ML concepts and MLOps; model validation in production.
• Familiarity with evaluation/observability tools.

Responsibilities

• Define evaluation strategies (golden sets, adversarial suites, regressions) and gates for quality, safety, latency, and cost.
• Establish rubric-based human reviews and calibrate annotators.
• Instrument LLM-as-judge with calibration and spot checks.
• Measure retrieval precision/recall, MRR/nDCG, and evidence provenance; detect hallucination.
• Test multi-step plans, tool usage, and idempotency for functions with side effects.
• Contract-test JSON schemas and structured outputs across services.
• Run token-aware load/soak tests; track p50/p95/p99, throughput, timeouts, and cost per task.
• Produce risk assessments and release recommendations in reports.

Skills

QA experience

API testing

Performance testing

Python/TypeScript

CI/CD

Cloud basics

ML concepts

Security testing

Evaluation tools

LLM tracing

Education

CS/IT degree

Tools

LangSmith

Weights & Biases

RAGAS

TruLens

Promptfoo

Guardrails/LlamaGuard

Presidio

OpenTelemetry LLm traces

Job description

NCS PTE. LTD. in Singapore is seeking an AI Tester to lead end-to-end evaluation of AI applications, including LLM features, RAG systems, and multi-agent workflows.

You will own test design, execution, and reporting across offline, pre-prod, and in-prod stages, and collaborate with product, data, and platform teams. The role focuses on non-functional testing, security, observability, and CI/CD integration, with responsibilities spanning test automation, ROI metrics, and clear risk-informed