Remote_QA tester-AI Agentic

Themesoft Inc; Denver (CO); Remote; USD 90,000 - 140,000; Full time

Job summary

The AI Agentic Tester role at Themesoft Inc. in Denver focuses on validating Generative AI and AI-based applications, including multi-agent workflows, RAG outputs, and prompt engineering.

You will ensure accuracy, relevance, and reliability across end-to-end AI use cases. Required are 3–8 years of QA experience, strong API testing skills, and proficiency in Python, Selenium/Playwright, and PyTest, with knowledge of Azure OpenAI, AWS Bedrock, or Gemini.

Qualifications

• 3-8 years of QA/Test Engineering experience.
• Experience testing Generative AI, LLM, or AI-based applications.
• Knowledge of AI Agents, Agentic AI, RAG, and prompt engineering.
• Hands-on experience with API testing (Postman, Swagger, REST APIs).
• Experience with automation frameworks (Selenium, Playwright, PyTest).
• Proficiency in Python.
• Understanding of AI evaluation metrics and testing methodologies.
• Knowledge of Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms.

Responsibilities

• Validate AI agent workflows against business requirements.
• Test end-to-end AI use cases, including multi-agent interactions.
• Verify task completion accuracy and output quality.
• Test agent orchestration, tool calling, and workflow execution.
• Evaluate LLM responses for accuracy, relevance, and consistency.
• Test prompt engineering effectiveness and response quality.
• Validate Retrieval-Augmented Generation (RAG) outputs.
• Measure hallucination rates and response reliability.

Skills

QA/Test Engineering

Generative AI testing

AI Agents / Agentic AI / RAG

Prompt engineering

API testing

Automation frameworks

Python

AI evaluation metrics

Azure OpenAI / AWS Bedrock / Gemini

Tools

Postman

Swagger

REST APIs

Selenium

Playwright

PyTest

Job description

Role: AI Agentic Tester

Must have- Experience testing Generative AI, LLM, or AI-based applications.

Keyword:

Skills: Modern Functional Quality Engineering Experience Required: 2-4 Years

PFB the responsibilities and skills needed:

Functional Testing

• Validate AI agent workflows against business requirements.
• Test end-to-end AI use cases, including multi-agent interactions.
• Verify task completion accuracy and output quality.
• Test agent orchestration, tool calling, and workflow execution.
• Evaluate LLM responses for accuracy, relevance, and consistency.
• Test prompt engineering effectiveness and response quality.
• Validate Retrieval-Augmented Generation (RAG) outputs.
• Measure hallucination rates and response reliability.

Required Skills

• 3-8 years of QA/Test Engineering experience.
• Experience testing Generative AI, LLM, or AI-based applications.
• Knowledge of AI Agents, Agentic AI, RAG, and prompt engineering.
• Hands-on experience with API testing (Postman, Swagger, REST APIs).
• Experience with automation frameworks (Selenium, Playwright, PyTest).
• Proficiency in Python.
• Understanding of AI evaluation metrics and testing methodologies Knowledge of Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms