GenAI QA Tester - Remote (Albany)

Cloud & Things Inc; City of Albany (NY); Remote; USD 65,000 - 79,000; Full time

Job summary

Cloud and Things is seeking an AI Tester to support a NYS Medicaid MECM modernization program in Albany. The role involves testing AI and generative AI solutions, including conversational AI and RAG, to ensure alignment with requirements and safeguards.

Remote options are available and the position is an exempt, 12-month engagement with a W2 hourly rate. The candidate will develop test plans, execute evaluation scenarios, and report findings with clear readiness recommendations to stakeholders.

Qualifications

• 2-5 years of hands-on experience as an AI Tester, including testing and quality assurance of AI solutions.
• Strong analytical skills and ability to recognize unexpected AI behavior.
• Experience working in regulated or public-sector environments.

Responsibilities

• Develop and execute test plans, test cases, and evaluation scenarios for conversational AI, RAG, agentic AI, and other GenAI solutions.
• Assess AI outputs for accuracy, relevance, consistency, completeness, and alignment with functional, business, and technical requirements.
• Perform negative, edge-case, and adversarial testing to identify unexpected AI behavior and failure modes.
• Validate content retention, guardrails, and overall solution quality; document findings and coordinate defect resolution.
• Track test results and quality metrics and communicate clear readiness recommendations to project stakeholders.

Skills

AI testing

QA testing

Analytical skills

Regulated environments

Tools

Jira

Job description

Cloud and Things is seeking an AI Tester to support a NYS Medicaid MECM modernization program in Albany. The role involves testing AI and generative AI solutions, including conversational AI and RAG, to ensure alignment with requirements and safeguards.

Remote options are available and the position is an exempt, 12-month engagement with a W2 hourly rate. The candidate will develop test plans, execute evaluation scenarios, and report findings with clear readiness recommendations to stakeholders.