Software Quality Assurance Engineer – GenAI / LLM

D L RESOURCES PTE LTD; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

D L RESOURCES PTE LTD is seeking a Senior GenAI Quality Engineer / Solution Analyst to design, analyse, test and validate production-grade GenAI, LLM, RAG and Agentic AI applications within a complex enterprise environment. This role blends software quality engineering with AI evaluation and solution analysis, spanning discovery to release.

Role emphasizes end-to-end testing of UI, REST APIs, backends and enterprise integrations, plus non-deterministic AI evaluation, safety testing, and

Qualifications

• 5-8 years of experience in Software Quality Engineering, Test Engineering, Test Automation or similar hands-on testing roles.
• Strong experience testing complex enterprise applications across web apps, APIs, backends and integrations.

Responsibilities

• Define and execute end-to-end quality engineering and test strategies for GenAI, LLM, RAG and agentic AI workloads.
• Perform GenAI/LLM testing covering response quality, grounding, faithfulness, relevance, hallucination risk and safe failure behaviour.
• Test automation development to reduce regression time and release cycle time.
• Validate Observability and troubleshooting data across logs, traces, APIs and data flows.

Skills

GenAI Testing

LLM Testing

API Testing

Test Automation

Python

Java

JavaScript

TypeScript

SDET

Quality engineering

Tools

Playwright

Cypress

Selenium

pytest

REST Assured

Postman

Job description

Primary Focus: Software Quality Assurance / Software Testing / Test Automation - GenAI, LLM & Agentic AI
Secondary Exposure: Solution Analysis / Technology Solution Design / Enterprise Integration
Domain / Project: Global Markets, Capital Markets Banking Technology & Market Risk Technology

Role Overview

We are looking for a Senior GenAI Quality Engineer / Solution Analyst to design, analyse, test and validate production-grade Generative AI (GenAI), Large Language Model (LLM), RAG and Agentic AI applications within a complex enterprise environment.

This is not a traditional manual QA or software testing role.

The role combines:

• Software Quality Engineering

• GenAI / LLM Testing & Evaluation

• Agentic AI / AI Agent Testing

• UI & API Testing

• Test Automation

• Solution Analysis

• Enterprise Integration Testing

• Observability & Troubleshooting

You will work across discovery, solution design, development, testing and release, translating business requirements into clear application behaviours and validating end-to-end application quality across user interfaces, APIs, data flows, LLMs, RAG components, AI agents and enterprise integrations.

Key Responsibilities

GenAI / LLM Quality Engineering

• Define and execute end-to-end quality engineering and test strategies covering:

  • Web / UI workflows

  • REST APIs

  • Backend services

  • Enterprise integrations

  • GenAI applications

  • LLM workflows

  • RAG pipelines

  • Agentic AI / AI Agent interfaces

• Perform GenAI / LLM testing and evaluation covering:

  • Response quality

  • Task completion

  • Grounding

  • Faithfulness

  • Relevance

  • Consistency

  • Citation accuracy

  • Hallucination risk

  • Safe failure behaviour

• Test non-deterministic / probabilistic AI systems using:

  • Evaluation datasets

  • Repeat testing

  • Quality thresholds

  • Acceptance criteria

  • Regression evaluation

• Validate RAG / Retrieval-Augmented Generation solutions, including retrieval quality, grounding and response accuracy.

Agentic AI / AI Agent Testing

Test end-to-end Agentic AI and AI Agent workflows, including:

• Multi-turn conversations

• Context handling

• Agent planning

• Tool selection

• Tool calling / function calling

• Tool inputs and outputs

• State transitions

• Memory and state

• Human-in-the-loop approvals

• Handoffs

• Retries

• Timeouts

• Fallback behaviour

• Error recovery

• Termination conditions

• Partial failures

Validate that AI agents behave correctly across both successful and failure scenarios.

Software & API Quality Engineering

Perform:

• Functional Testing

• Integration Testing

• API Testing

• Regression Testing

• Exploratory Testing

• Negative Testing

• Resilience Testing

• Basic Performance Testing

• End-to-End Testing

Design comprehensive REST API tests covering:

• API contracts

• Authentication

• Authorisation

• Input validation

• Error handling

• Idempotency

• Rate limits

• Downstream system failures

Test web application behaviour across browsers and realistic end-user journeys, including:

• Loading states

• Interrupted sessions

• Error messages

• Feedback capture

• Accessibility fundamentals

Test Automation

Develop and maintain risk-based test automation that reduces:

• Regression testing time

• Manual testing effort

• Release cycle time

• Production risk

Use automation frameworks and tools such as:

• Playwright

• Cypress

• Selenium

• pytest

• REST Assured

• Postman

• Equivalent UI / API automation frameworks

Apply pragmatic automation principles by prioritising stable, high-value and frequently executed test scenarios.

GenAI Evaluation & AI Safety Testing

Validate LLM and GenAI applications for:

• Grounded responses

• Hallucinations

• Retrieval quality

• Citation accuracy

• Prompt behaviour

• Prompt injection

• Unsupported requests

• Restricted content handling

• Safe failure behaviour

• Adversarial scenarios

Support AI evaluation / LLM evaluation using appropriate evaluation datasets, quality metrics and repeatable evaluation approaches.

Exposure to AI Red Teaming / Adversarial Testing would be advantageous.

Observability & Troubleshooting

Use application and GenAI observability to identify the source of defects across:

• Application

• LLM / Model

• RAG / Retrieval

• Data

• API / Integration

• Platform

Analyse:

• Logs

• Distributed traces

• API requests / responses

• Payloads

• Network calls

• Database records

• Agent execution traces

Exposure to observability and LLM evaluation tools such as:

• Langfuse

• LangSmith

• OpenTelemetry

• Elastic / Elasticsearch

• Splunk

is advantageous.

Solution Analysis & Design

The role also acts as a hands‑on Solution Analyst for GenAI applications.

Responsibilities include:

• Partner with product owners, business users, architects, engineers and GenAI specialists during discovery and solution design.

• Analyse proposed GenAI use cases and determine whether the requirement should use:

  • Conventional application logic

  • Deterministic business rules

  • Search / retrieval

  • RAG

  • Workflow automation

  • Agentic AI

  • Human approval

• Translate business requirements into:

  • Functional requirements

  • End-to-end solution flows

  • User journeys

  • Acceptance criteria

  • Interface behaviour

  • Decision rules

  • Non-functional requirements

• Map interactions across:

  • User Interfaces

  • APIs

  • LLMs / Models

  • Prompts

  • RAG / Retrieval components

  • Enterprise data sources

  • AI Agent tools

  • Downstream enterprise systems

• Analyse solution design trade-offs involving:

  • Quality

  • Complexity

  • Cost

  • Latency

  • Security

  • Data access

  • Maintainability

  • Operational risk

• Identify missing controls, integration assumptions, ownership gaps, failure scenarios and operational risks before development begins.

• Support the design of:

  • Human-in-the-loop approval

  • Fallback flows

  • Escalation

  • Exception handling

Solution Documentation

Produce practical technical and functional artefacts including:

• Process Flows

• Sequence Diagrams

• Context Diagrams

• Interface Specifications

• Decision Tables

• User Stories

• Acceptance Criteria

• Test Scenarios

• Traceability Documentation

Maintain traceability across:

Business Requirement → Solution Design → Implementation → Test / Evaluation Scenario → Release Evidence

Release Quality & Governance

Create and maintain:

• Test scenarios

• Test datasets

• Reusable regression scenarios

• Test evidence

• Defect reports

• Quality metrics

• Release quality reports

Provide evidence-based release recommendations identifying:

• Known defects

• Known limitations

• Residual risks

• Quality concerns

• Areas requiring production monitoring

Core Requirements

Experience

• 5-8 years of experience in Software Quality Engineering, Test Engineering, Test Automation, SDET or similar hands‑on software testing roles.

• Strong experience testing complex enterprise applications.

• Strong experience testing:

  • Web applications

  • REST APIs

  • Backend services

  • Enterprise integrations

Test Automation / Programming

Hands‑on experience with one or more of:

• Playwright

• Cypress

• Selenium

• pytest

• REST Assured

• Postman

• Equivalent automation frameworks

Working programming knowledge of:

• Python

• Java

• JavaScript

• TypeScript

Candidates should be capable of developing, reviewing and troubleshooting test automation.

Software Engineering / DevOps

Experience with:

• Git

• Pull Requests

• CI/CD

• Automated Testing

• Test Reporting

• Defect Management

Experience validating distributed systems including:

• Asynchronous Processing

• Queues

• Batch Processing

• APIs

• Downstream Dependencies

• Enterprise Integrations

GenAI / LLM Requirements

Practical understanding of:

• Generative AI / GenAI

• Large Language Models / LLM

• LLM Evaluation

• LLM Testing

• Retrieval-Augmented Generation / RAG

• RAG Evaluation

• Agentic AI

• AI Agents

• Multi-Agent Workflows

• Prompts / Prompt Engineering

• Context Windows

• Embeddings

• Tool Calling

• Agent Memory & State

• LLM Observability

Candidates should understand how GenAI applications differ from conventional deterministic software and how to validate probabilistic AI behaviour.

Security & Risk Testing

Understanding of software and GenAI security fundamentals including:

• Access Control

• Authentication / Authorisation

• Sensitive Data Handling

• Input Validation

• Auditability

• Prompt Injection

• AI Safety Testing

• Adversarial Testing

Nice to Have

Experience with:

• Banking / Financial Services

• Regulated enterprise environments

• Contract Testing

• Service Virtualisation

• Synthetic Monitoring

• Performance Testing

• AI Red Teaming

• Accessibility Testing / WCAG

• Kubernetes

• OpenShift

• AWS

• Containerised Application Deployment

Key Domain / Technical Skills

1. Software Quality Engineering, API Testing & Test Automation

2. GenAI / LLM Evaluation, RAG & Agentic AI Testing

3. Solution Analysis, Observability & Enterprise Integration

Key Search Keywords

GenAI Quality Engineer,AI Quality Engineer,LLM Quality Engineer,Generative AI Testing,GenAI Testing,LLM Testing,LLM Evaluation,AI Evaluation,Agentic AI Testing,AI Agent Testing,RAG Testing,RAG Evaluation,Retrieval-Augmented Generation,Software Quality Engineering,Quality Engineering,Software QA,Test Automation,SDET,Automation Testing,API Testing,REST API Testing,UI Testing,Integration Testing,Regression Testing,End-to-End Testing,Playwright,Cypress,Selenium,pytest,REST Assured,Postman,Python,Java,JavaScript,TypeScript,CI/CD,Git,Prompt Testing,Prompt Injection,Hallucination Testing,Grounding,Faithfulness,AI Safety Testing,Adversarial Testing,AI Red Teaming,Langfuse,LangSmith,OpenTelemetry,Elastic,Splunk,Observability,Distributed Systems,Kubernetes,OpenShift,Solution Analysis