Generative AI QA Engineer — Bedrock & Conversational Testing

NTT DATA, Europe & LATAM, Branch in USA, Inc.

United States

Remote

USD 28,000 - 34,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

NTT DATA is seeking an AI Testing Engineer to define and execute the quality strategy for Generative AI applications and AI agents used in customer service scenarios. You will design evaluation plans, datasets, and regression suites for multi-turn conversations and ensure robust testing across deterministic and non-deterministic outputs.

This role requires hands-on automation, REST API testing, experience with Amazon Bedrock, and strong English proficiency.

Qualifications

  • 5+ years in software testing, quality engineering, or test automation.
  • 2+ years of experience with Amazon Bedrock.
  • Understanding of Generative AI and testing non-deterministic systems.
  • Experience designing test scenarios for conversational apps, chatbots, or AI agents.
  • Experience testing REST APIs and backend services.
  • Proficiency in English.

Responsibilities

  • Define testing and evaluation strategy for Generative AI apps and agents.
  • Create evaluation datasets, golden datasets, and regression suites.
  • Design single-turn and multi-turn conversational test cases.
  • Evaluate AI outputs for accuracy, relevance, safety, and compliance.
  • Build automated evaluation pipelines and integrate AI-quality checks into CI/CD.
  • Produce quality metrics, evaluation reports, dashboards, and trend analyses.

Skills

Test automation
Amazon Bedrock
LLM evaluation
REST API testing
Git & CI/CD
English proficiency

Education

Bachelor’s degree in Computer Science, AI, ML, or related field

Tools

LangSmith
TruLens
Promptfoo
DeepEval
RAGAS

Job description

NTT DATA is seeking an AI Testing Engineer to define and execute the quality strategy for Generative AI applications and AI agents used in customer service scenarios. You will design evaluation plans, datasets, and regression suites for multi-turn conversations and ensure robust testing across deterministic and non-deterministic outputs.

This role requires hands-on automation, REST API testing, experience with Amazon Bedrock, and strong English proficiency.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Testing Engineer, Amazon Bedrock
AI Testing Engineer, Amazon Bedrock

NTT DATA, Europe & LATAM, Branch in USA, Inc. • United States

Remote
USD 28,000 - 34,000
QA Automation Engineer - GenAI / LLM Testing (Healthcare)
QA Automation Engineer - GenAI / LLM Testing (Healthcare)

Luxoft • United States

Remote
USD 110,000 - 150,000
Generative AI Engineer - AWS Bedrock & SageMaker
Generative AI Engineer - AWS Bedrock & SageMaker

Tata Consultancy Services Limited • Malvern

On-site
USD 120,000 - 140,000
Discretionary Annual Incentive
Comprehensive Medical Coverage
Parental Leaves
+5
GMP Compliance & Equipment Qualification Specialist
GMP Compliance & Equipment Qualification Specialist

TechDigital Group • Summit (NJ)

On-site
USD 80,000 - 110,000
Remote Generative AI QA Tester
Remote Generative AI QA Tester

Volto Consulting • Colorado

Remote
USD 90,000 - 130,000
GenAI Solutions Architect - AWS Bedrock & AgentCore
GenAI Solutions Architect - AWS Bedrock & AgentCore

Tenth Revolution Group • United States

On-site
USD 140,000 - 190,000
QA Automation Analyst - GenAI-Driven Testing
QA Automation Analyst - GenAI-Driven Testing

Robotic Technologies Automation • Boston (MA)

On-site
USD 98,000 - 139,000
QA Analyst - AI-Driven Testing & Data Validation
QA Analyst - AI-Driven Testing & Data Validation

Quantum Technologies. LLC • New Jersey

On-site
USD 65,000 - 110,000
Lead QA Engineer for AI-Driven Testing & Automation
Lead QA Engineer for AI-Driven Testing & Automation

Genpact Ltd. • United States

Remote
USD 60,000 - 75,000
GenAI & AI Testing Engineer - QA & Automation
GenAI & AI Testing Engineer - QA & Automation

Ethereum Technologies LLC • Irving (TX)

On-site
USD 90,000 - 130,000