AI Tester

NCS PTE. LTD.; Singapore; On-site; SGD 72,000 - 120,000; Full time

Job summary

NCS PTE. LTD. in Singapore is seeking an AI Tester to lead end-to-end evaluation of AI applications, including LLM features, RAG systems, and multi-agent workflows.

You will own test design, execution, and reporting across offline, pre-prod, and in-prod stages, and collaborate with product, data, and platform teams. The role focuses on non-functional testing, security, observability, and CI/CD integration, with responsibilities spanning test automation, ROI metrics, and clear risk-informed

Qualifications

• 3+ years of software testing/QA with API and performance testing.
• Experience with CI/CD and version control.
• Understanding of ML concepts and MLOps; model validation in production.
• Familiarity with evaluation/observability tools.

Responsibilities

• Define evaluation strategies (golden sets, adversarial suites, regressions) and gates for quality, safety, latency, and cost.
• Establish rubric-based human reviews and calibrate annotators.
• Instrument LLM-as-judge with calibration and spot checks.
• Measure retrieval precision/recall, MRR/nDCG, and evidence provenance; detect hallucination.
• Test multi-step plans, tool usage, and idempotency for functions with side effects.
• Contract-test JSON schemas and structured outputs across services.
• Run token-aware load/soak tests; track p50/p95/p99, throughput, timeouts, and cost per task.
• Produce risk assessments and release recommendations in reports.

Skills

QA experience

API testing

Performance testing

Python/TypeScript

CI/CD

Cloud basics

ML concepts

Security testing

Evaluation tools

LLM tracing

Education

CS/IT degree

Tools

LangSmith

Weights & Biases

RAGAS

TruLens

Promptfoo

Guardrails/LlamaGuard

Presidio

OpenTelemetry LLm traces

Job description

NCS is a leading AI Tech Services company. With a 15,000-strong team across the Asia Pacific, NCS scales its platforms and capabilities to provide clients with greater agility and AI expertise across a range of Industries. Embracing a strong ecosystem of global partners, NCS transforms technology services delivery combining AI with digital resilience to drive real business impact. NCS is a subsidiary of the Singtel Group.

As an AI Tester, you will lead the end-to-end evaluation of AI applications—LLM features, RAG systems, and multi-agent workflows—to ensure they meet business outcomes, safety requirements, and platform standards. Own test design, execution, and reporting across offline, pre- prod, and in-prod stages, integrating with CI/CD and working closely with product, data, and platform teams.

What will you do?

AI/LLM Evaluation & Test Design

• Define evaluation strategies (golden sets, adversarial suites, regressions), pass/fail gates, and SLOs for quality, safety, latency, and cost.
• Establish rubric-based human reviews (usefulness, faithfulness, safety, clarity) and calibrate annotators.
• Instrument LLM-as-judge where appropriate with calibration and spot checks.

RAG, Retrieval, & Grounding

• Measure retrieval precision/recall, MRR/nDCG, and answer faithfulness to sources; detect hallucination and citation errors.
• Test chunking, prompt templates, filters, and policy chains; monitor stale/poisoned content.

Agentic & Tool-Use Scenarios

• Validate multi-step plans, tool selection, error recovery, retries, and idempotency for functions with side effects.
• Contract-test JSON schemas and structured outputs across services.

Non-Functional, Performance & Cost

• Run token-aware load/soak tests (context length, temperature, batching); track p50/p95/p99, throughput, timeouts, cache hit rate, and cost per successful task.
• Recommend optimizations (prompt/policy changes, retrieval tweaks, caching).

Security, Privacy & Safety

• Red-team for prompt injection, data exfiltration, indirect injections via retrieved content; validate guardrails pre/post inference.
• Enforce PII controls, data-residency, and compliance checks; align with organizational security testing practices.

Observability & CI/CD Integration

• Implement prompt/dataset/version lineage and trace-based evals; automate in CI (pre-merge golden tests, nightly adversarials) with canary/A-B in prod and rollback criteria.
• Produce clear, decision-ready reports with risk assessments and release recommendations.

Project Delivery & Collaboration

• Analyze requirements, enhance test plans with additional cases, prepare environments (including cloud), execute tests per plan, and drive defect resolution.
• Provide regular status updates; manage test activities to schedule; support SIT/UAT and production readiness.

Performance, API & Platform Testing (Carry-over)

• Execute API, performance, and load testing for microservices/web services that underpin AI features; integrate automated testing into CI/CD.

The ideal candidate should possess:

• 3+ years in software testing/QA with strong test methodology and tooling; hands- on API testing and performance testing.
• Programming familiarity (e.g., Python/TypeScript) and experience with CI/CD and version control.
• Cloud basics (AWS/Azure/GCP) and microservices fundamentals.
• Degree/Diploma in CS/IT or equivalent.
• Understanding of ML concepts and MLOps; experience with model validation and monitoring in production.
• Experience with AI-specific security testing and vulnerability assessment.
• Familiarity with evaluation/observability tools (any of): LangSmith, Weights & Biases, RAGAS, TruLens, Promptfoo, DeepEval, Guardrails/LlamaGuard, Presidio; plus OpenTelemetry-style LLM traces.
• Practical exposure to Azure OpenAI/Bedrock/Vertex and model gateways; quota & token accounting know-how.
• Modern automation frameworks (e.g., Playwright, Cypress, Selenium), API test tools (Postman/REST Assured), performance tools (k6/JMeter), and CI/CD integration.
• Data evaluation pipelines for RAG (embedding validation, filtering, drift detection).
• Experience testing copilots/agents for BSS/OSS, NOC analytics, and enterprise care; ability to tie eval KPIs to CSAT, AHT, FCR, MTTR.

We are driven by our AEIOU beliefs—Adventure, Excellence, Integrity, Ownership, and Unity—and we seek individuals who embody these values in both their professional and personal lives. We are committed to our Impact: Valuing our clients, Growing our people, and Creating our future.

Together, we make the extraordinary happen.

Learn more about us at ncs.co and visit our LinkedIn career site.