Sr GenAI Quality & Solution Analyst (Banking, 1-year renewable contract)

EVOLUTION RECRUITMENT SOLUTIONS PTE. LTD.; Singapore; Vor Ort; SGD 90.000 - 120.000; Vollzeit

Zusammenfassung

Evolution Recruitment Solutions Pte. Ltd. is seeking an experienced Quality Engineer to design and execute end-to-end test strategies across UI, backend APIs, and integrations.

You will validate GenAI applications for task completion, accuracy, grounding, and safe failure handling. The role covers functional, integration, regression, exploratory, and performance testing with a focus on automation and release governance.

Qualifikationen

• 5–8 years of hands-on experience in software quality engineering or similar.
• Strong experience testing web apps, backend services, REST APIs and enterprise integrations.
• Hands-on API testing and automation with modern tools.
• Proficiency in Java, Python, JavaScript or TypeScript for test automation.
• Familiarity with Git, CI/CD pipelines, test reporting and defect management.

Aufgaben

• Design and execute end-to-end test strategies for UI, APIs, and integrations.
• Validate GenAI applications for accuracy, grounding, and safe failure handling.
• Automate high-value test scenarios to reduce manual effort and risk.
• Analyze logs, traces, payloads, and DB records to identify root causes.
• Collaborate with product, Eng, security, and GenAI teams to define outcomes.

Kenntnisse

Test analysis

Distributed systems testing

Risk assessment

Requirements review

Stakeholder communication

Tools

Postman

REST Assured

pytest

Playwright

Cypress

Selenium

Jobbeschreibung

Please note that visa sponsorship is not available at this time.

Key Responsibilities

• Design and execute end-to-end test strategies covering user interfaces, backend APIs, integrations, data flows, and agentic AI behaviour.
• Validate GenAI applications for task completion, response accuracy, relevance, grounding, consistency, citation quality, and safe failure handling.
• Test agentic workflows, including tool selection and execution, multi-turn context, state transitions, retries, timeouts, approval checkpoints, handoffs, and recovery from failures.
• Perform functional, integration, regression, exploratory, negative, resilience, and basic performance testing across application layers.
• Develop and maintain API tests covering authentication, authorisation, validation, error handling, idempotency, rate limits, and downstream dependencies.
• Automate high-value test scenarios to improve testing efficiency, reduce manual effort, and minimise production risks.
• Analyse logs, traces, request and response payloads, network calls, and database records to identify defects and determine root causes.
• Establish reusable test scenarios, test data, traceability, and evidence to support enterprise release governance and quality reporting.
• Provide evidence-based release recommendations, highlighting known limitations, residual risks, and areas requiring further monitoring.
• Collaborate with product owners, business users, architects, engineers, security teams, and GenAI specialists to define business needs, user journeys, and expected outcomes.
• Translate business requirements into functional requirements, end-to-end solution flows, interface behaviours, business rules, acceptance criteria, and non-functional requirements.
• Analyse solution options and recommend appropriate approaches involving GenAI, retrieval-augmented generation (RAG), workflow orchestration, deterministic application logic, and human approval.
• Map interactions across user interfaces, APIs, language models, prompts, retrieval components, enterprise data sources, agent tools, and downstream systems.
• Identify design gaps, integration assumptions, ownership issues, security concerns, failure scenarios, and operational risks before development begins.
• Facilitate solution design reviews and document findings through process flows, sequence diagrams, interface specifications, decision tables, and solution documentation.
• Define measurable success criteria covering business outcomes, user experience, functional correctness, response quality, latency, reliability, and safe failure.
• Maintain traceability from business requirements through solution design, implementation, test scenarios, and release evidence.

Key Requirements

• 5–8 years of hands-on experience in software quality engineering, test engineering, or a similar role involving complex applications.
• Strong experience testing web applications, backend services, REST APIs, and enterprise integrations.
• Hands-on experience with API testing and automation tools such as Postman, REST Assured, pytest, Playwright, Cypress, Selenium, or equivalent.
• Working knowledge of Java, Python, JavaScript, or TypeScript to develop, review, and troubleshoot test automation.
• Strong test analysis skills, including requirements review, risk assessment, boundary analysis, negative testing, and traceability.
• Experience testing distributed systems and integrations, including asynchronous processing, queues, batch jobs, and downstream dependencies.
• Ability to investigate application issues using logs, traces, network calls, payloads, and database records.
• Familiarity with Git, pull requests, CI/CD pipelines, test reporting, and defect management tools.
• Understanding of security and privacy testing fundamentals, including access control, sensitive data handling, input validation, and auditability.
• Experience analysing complex applications across business processes, user journeys, APIs, data flows, and system integrations.
• Ability to facilitate requirements discussions and translate ambiguous busines needs into clear functional requirements, acceptance criteria, and solution behaviours.
• Experience producing analysis artefacts such as process flows, sequence diagrams, context diagrams, interface specifications, decision tables, and user stories.
• Understanding of application architecture concepts, including synchronous and asynchronous integrations, event flows, authentication, authorisation, system boundaries, and failure handling.
• Ability to assess solution alternatives and explain trade-offs involving quality, cost, performance, security, operational support, and delivery complexity.
• Practical understanding of LLM-based applications, RAG, prompts, context windows, embeddings, and tool-using agents.
• Ability to evaluate probabilistic AI systems using evaluation datasets, repeated test runs, quality thresholds, and evidence-based acceptance criteria rather than relying solely on exact text matching.
• Experience validating grounded responses, citations, retrieval quality, hallucination risks, prompt injection resistance, and safe handling of restricted or unsupported requests.
• Understanding of agent planning and execution, tool calls, memory and state management, human approval checkpoints, fallback behaviour, and terminationconditions.
• Familiarity with observability and evaluation tools such as Langfuse, LangSmith, OpenTelemetry, Elastic, Splunk, or equivalent.
• Strong analytical, communication, and stakeholder management skills, with the confidence to challenge unclear requirements, weak designs, and premature release decisions.
• Ability to work effectively in a fast-paced environment where requirements and GenAI behaviour continue to evolve.

Nice to Have

• Experience in banking, financial services, or other regulated enterprise environments.
• Exposure to contract testing, service virtualisation, synthetic monitoring, or performance testing.
• Familiarity with Kubernetes, OpenShift, AWS-hosted services, or containerised deployments.
• Accessibility testing experience and familiarity with WCAG-based checks.
• Experience developing quality dashboards covering release risks, defect trends, flaky tests, and testing cycle time.
• Exposure to red teaming or adversarial testing of GenAI applications.