Senior GenAI QA & Solution Analyst — LLM/RAG Testing

NTT SINGAPORE PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Part time

Job summary

NTT SINGAPORE PTE. LTD. seeks a Senior GenAI Quality Engineer to analyse, test and validate enterprise Generative AI apps in a regulated financial-services setting.

The role focuses on UI/API automation and hands-on testing of production LLM/RAG systems, translating ambiguous requirements into concrete flows, criteria and release evidence. You will define end-to-end risk-based test strategies for web UI, REST APIs, and AI components, ensuring accuracy, grounding and safe failure across

Qualifications

• 4+ years of hands-on software quality engineering, SDET or test-automation experience.
• Recent experience testing a real production or enterprise GenAI, LLM, RAG, chatbot or agentic application.
• Strong UI and REST API automation experience.
• Hands-on Playwright, Cypress or Selenium experience.
• Hands-on Postman, REST Assured, PyTest or equivalent API automation.
• Working knowledge of Python, Java, JavaScript or TypeScript.
• Experience validating hallucination, grounding, factuality, relevance, citations and multi-turn context.
• Experience testing retrieval quality, document ingestion, chunking, embeddings, reranking or vector-search behaviour.
• Experience with repeat-run evaluation, golden datasets and threshold-based acceptance.
• Experience analysing application logs, model traces, API payloads and database records.
• Strong requirements analysis, risk assessment, negative testing and traceability.
• Experience with Git, pull requests, CI/CD, test reporting and defect-management tools.
• Strong written and verbal stakeholder communication.

Responsibilities

• Define end-to-end, risk-based test strategies covering web UI, REST APIs, backend services, databases, enterprise integrations and GenAI components.
• Validate LLM and RAG responses for task completion, grounding, relevance, factual accuracy, consistency, citations, hallucination risk and safe failure.
• Create evaluation datasets, golden test sets, quality thresholds and repeat-run test approaches for non-deterministic AI outputs.
• Test conversational and agentic behaviour, including multi-turn context, memory, tool selection, tool inputs and outputs, state transitions and termination conditions.
• Validate retries, timeouts, handoffs, human-approval checkpoints, fallback behaviour and recovery from partial failures.
• Perform functional, integration, regression, exploratory, negative, resilience, security and basic performance testing.
• Design API tests covering authentication, authorisation, contracts, validation, error handling, idempotency, rate limits and downstream failures.
• Develop and maintain UI and API automation using Playwright, Cypress, Selenium, PyTest, REST Assured, Postman or equivalent tools.
• Analyse logs, traces, network calls, payloads and database records to isolate defects across components.
• Validate prompt-injection resistance, restricted-data handling, access controls, auditability and safe responses.

Skills

GenAI testing

UI automation

API automation

Playwright

Cypress

Python

JavaScript/TypeScript

SDET

CI/CD

Education

Bachelor's degree in Computer Science or equivalent

Tools

Playwright

Cypress

Selenium

Postman

REST Assured

PyTest

Job description

NTT SINGAPORE PTE. LTD. seeks a Senior GenAI Quality Engineer to analyse, test and validate enterprise Generative AI apps in a regulated financial-services setting.

The role focuses on UI/API automation and hands-on testing of production LLM/RAG systems, translating ambiguous requirements into concrete flows, criteria and release evidence. You will define end-to-end risk-based test strategies for web UI, REST APIs, and AI components, ensuring accuracy, grounding and safe failure across