AI Agentic Tester — QA for LLM & AI Workflows (Remote)

TALENT Software Services; Denver (CO); On-site; USD 90,000 - 140,000; Full time

Job summary

TALENT Software Services is seeking an AI Agentic Tester for a remote Denver-based role. You will validate AI agent workflows, test end-to-end AI use cases, and ensure output quality across multi-agent interactions.

The ideal candidate has 3–8 years of QA/testing experience, strong Python skills, API testing experience, and familiarity with AI platforms such as Azure OpenAI, AWS Bedrock, or Gemini to ensure robust AI performance.

Qualifications

• 2-8 years of QA/Test Engineering experience.
• Experience testing Generative AI, LLM, or AI-based applications.
• Knowledge of AI Agents, Agentic AI, RAG, and prompt engineering.
• Hands-on API testing (Postman, Swagger, REST APIs).
• Experience with automation frameworks (Selenium, Playwright, PyTest).
• Proficiency in Python.
• Understanding of AI evaluation metrics and testing methodologies.
• Knowledge of Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms.

Responsibilities

• Validate AI agent workflows against business requirements.
• Test end-to-end AI use cases, including multi-agent interactions.
• Verify task completion accuracy and output quality.
• Test agent orchestration, tool calling, and workflow execution.
• Evaluate LLM responses for accuracy, relevance, and consistency.
• Test prompt engineering effectiveness and response quality.
• Validate Retrieval-Augmented Generation (RAG) outputs.
• Measure hallucination rates and response reliability.

Skills

QA/Test Engineering

Generative AI testing

AI Agents / Agentic AI / RAG

Prompt engineering

Python

API testing

Automation frameworks

AI platforms knowledge

Tools

Postman

Swagger

REST APIs

Selenium

Playwright

PyTest

Job description

TALENT Software Services is seeking an AI Agentic Tester for a remote Denver-based role. You will validate AI agent workflows, test end-to-end AI use cases, and ensure output quality across multi-agent interactions.

The ideal candidate has 3–8 years of QA/testing experience, strong Python skills, API testing experience, and familiarity with AI platforms such as Azure OpenAI, AWS Bedrock, or Gemini to ensure robust AI performance.