Generative AI QA Engineer — Bedrock & Conversational Testing

NTT DATA, Europe & LATAM, Branch in USA, Inc.

United States

Remote

USD 28,000 - 34,000

Full time

6 days ago

Be an early applicant

Application generator

A complete application in a minute — tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

NTT DATA is seeking an AI Testing Engineer to define and execute the quality strategy for Generative AI applications and AI agents used in customer service scenarios. You will design evaluation plans, datasets, and regression suites for multi-turn conversations and ensure robust testing across deterministic and non-deterministic outputs.

This role requires hands-on automation, REST API testing, experience with Amazon Bedrock, and strong English proficiency.

Qualifications

5+ years in software testing, quality engineering, or test automation.
2+ years of experience with Amazon Bedrock.
Understanding of Generative AI and testing non-deterministic systems.
Experience designing test scenarios for conversational apps, chatbots, or AI agents.
Experience testing REST APIs and backend services.
Proficiency in English.

Responsibilities

Define testing and evaluation strategy for Generative AI apps and agents.
Create evaluation datasets, golden datasets, and regression suites.
Design single-turn and multi-turn conversational test cases.
Evaluate AI outputs for accuracy, relevance, safety, and compliance.
Build automated evaluation pipelines and integrate AI-quality checks into CI/CD.
Produce quality metrics, evaluation reports, dashboards, and trend analyses.