AI Tester

Cloud & Things Inc; City of Albany (NY); Remote; USD 65,000 - 79,000; Full time

Job summary

Cloud and Things is seeking an AI Tester to support a NYS Medicaid MECM modernization program in Albany. The role involves testing AI and generative AI solutions, including conversational AI and RAG, to ensure alignment with requirements and safeguards.

Remote options are available and the position is an exempt, 12-month engagement with a W2 hourly rate. The candidate will develop test plans, execute evaluation scenarios, and report findings with clear readiness recommendations to stakeholders.

Qualifications

• 2-5 years of hands-on experience as an AI Tester, including testing and quality assurance of AI solutions.
• Strong analytical skills and ability to recognize unexpected AI behavior.
• Experience working in regulated or public-sector environments.

Responsibilities

• Develop and execute test plans, test cases, and evaluation scenarios for conversational AI, RAG, agentic AI, and other GenAI solutions.
• Assess AI outputs for accuracy, relevance, consistency, completeness, and alignment with functional, business, and technical requirements.
• Perform negative, edge-case, and adversarial testing to identify unexpected AI behavior and failure modes.
• Validate content retention, guardrails, and overall solution quality; document findings and coordinate defect resolution.
• Track test results and quality metrics and communicate clear readiness recommendations to project stakeholders.

Skills

AI testing

QA testing

Analytical skills

Regulated environments

Tools

Jira

Job description

Back AI Tester
Other Area(s) Albany , New York W-2 hourly Nov 5, 2026

Our goal is to solve problems and deliver results for our clients. At Cloud and Things, you can be a part of transforming the public sector’s IT environment. Our team is on the forefront of helping to solve the government's most complex IT challenges. If you are seeking a role that offers the opportunity to work on rewarding projects, consider a career with Cloud and Things.

*This is an exempt position. Salary commensurate with experience*

Job Title: AI Tester
Location: Albany, NY (Remote Available)
Duration: 12 Months (with potential for extension)
W2 Hourly Rate: $47-$57/hour
Application Deadline: November 8, 2026

Overview

We are seeking an AI Tester who will support our NYS client on the Medicaid Eligibility and Client Management (MECM) modernization program. The consultant will test AI and generative AI solutions, including conversational AI, retrieval-augmented generation (RAG), and agentic AI, to ensure alignment with business, functional, and technical requirements. This role will assess output quality, validate safeguards, and provide testing metrics and readiness recommendations.

Duties

• Develop and execute test plans, test cases, and evaluation scenarios for conversational AI, RAG, agentic AI, and other GenAI solutions.
• Assess AI outputs for accuracy, relevance, consistency, completeness, and alignment with functional, business, and technical requirements.
• Perform negative, edge-case, and adversarial testing to identify unexpected AI behavior and failure modes.
• Validate content retention, guardrails, and overall solution quality; document findings and coordinate defect resolution.
• Track test results and quality metrics and communicate clear readiness recommendations to project stakeholders.

Mandatory Qualifications

• 2-5 years of hands-on experience as an AI Tester, including testing and quality assurance of AI solutions.

Desirable Qualifications

• Experience with software quality assurance and testing; AI/ML or generative AI testing experience preferred.
• Familiarity with large language models (LLMs) and AI evaluation techniques.
• Experience with test management and defect-tracking tools such as Jira.
• Strong analytical skills and ability to recognize unexpected AI behavior.
• Experience working in regulated or public-sector environments.

What We Offer

• Opportunities to lead impactful transformations in the public sector infrastructure.
• Engagement with high-profile government initiatives, supporting meaningful change.
• Professional development through exposure to innovative Agile practices and leadership collaboration.
• A collaborative environment that values ownership, initiative, and continuous learning.

Ready to make a difference?

We’re eager to connect with qualified candidates committed to delivering results and fostering excellence within client projects.

Cloud and Things complies with all applicable federal, state, and local laws regarding recruitment and hiring. All qualified applicants are considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected veteran status, or any other category protected by applicable federal, state, or local laws.