AI QA Engineer: Test AI Features & Prompt Crafting

PowerToFly; Charlotte (NC); On-site; USD 78,000 - 114,000; Full time

Benefits offered by this job

Medical

Dental

Vision

Parental leave

Paid time off

401(k) plan with employee and company"

Life insurance

Disability insurance

Discounted employee stock purchasepl

Tuition reimbursement

Job summary

Moody's is seeking a Quality Engineer to support Agile QA for software and AI-enabled products, focusing on manual, exploratory, regression, API, and integration testing within a collaborative team.

You will document defects, create test data and prompts for evaluating AI behavior, and contribute to improving QA processes while partnering with engineering, data science, and product teams to deliver reliable risk assessment tools.

Qualifications

• Basic understanding of software testing concepts across manual, exploratory, regression, API, and integration testing.
• Familiarity with APIs, databases, JSON, version control, and defect tracking tools.
• Basic understanding of AI concepts, large language models, and responsible AI practices.

Responsibilities

• Perform manual, exploratory, regression, API, and integration testing within an Agile team.
• Test features with AI/LLM components for accuracy, relevance, completeness, consistency, and instruction-following.
• Create test scenarios, prompts, and data for evaluating AI behavior and identifying issues such as hallucinations, incomplete responses, and inconsistent results.
• Apply basic prompt engineering and context engineering techniques during testing activities.
• Use AI-assisted tools under the guidance of senior team members.
• Assist with existing AI evaluation harnesses, test scripts, and regression checks to support quality validation efforts.
• Document defects with clear steps to reproduce, expected results, and actual results, and collaborate with engineers, developers, and product owners to investigate and resolve issues.
• Contribute to test documentation, reusable test cases, and the continuous improvement of quality assurance processes and standards.

Skills

Software testing concepts

Python basics

APIs & JSON

AI concepts & LLMs

Prompt engineering

Defect documentation

Collaborative problem solving

Education

Bachelor's degree in CS/SE or related

Tools

Codex

Claude Code

AI tooling CLI

Job description

Moody's is seeking a Quality Engineer to support Agile QA for software and AI-enabled products, focusing on manual, exploratory, regression, API, and integration testing within a collaborative team.

You will document defects, create test data and prompts for evaluating AI behavior, and contribute to improving QA processes while partnering with engineering, data science, and product teams to deliver reliable risk assessment tools.