AI Agentic Tester — QA for LLM & AI Workflows (Remote)

Talent Software Services, Inc.

Denver (CO)

Remote

USD 90,000 - 140,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

TALENT Software Services is seeking an AI Agentic Tester for a remote Denver-based role. You will validate AI agent workflows, test end-to-end AI use cases, and ensure output quality across multi-agent interactions.

The ideal candidate has 3–8 years of QA/testing experience, strong Python skills, API testing experience, and familiarity with AI platforms such as Azure OpenAI, AWS Bedrock, or Gemini to ensure robust AI performance.

Qualifications

  • 2-8 years of QA/Test Engineering experience.
  • Experience testing Generative AI, LLM, or AI-based applications.
  • Knowledge of AI Agents, Agentic AI, RAG, and prompt engineering.
  • Hands-on API testing (Postman, Swagger, REST APIs).
  • Experience with automation frameworks (Selenium, Playwright, PyTest).
  • Proficiency in Python.
  • Understanding of AI evaluation metrics and testing methodologies.
  • Knowledge of Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms.

Responsibilities

  • Validate AI agent workflows against business requirements.
  • Test end-to-end AI use cases, including multi-agent interactions.
  • Verify task completion accuracy and output quality.
  • Test agent orchestration, tool calling, and workflow execution.
  • Evaluate LLM responses for accuracy, relevance, and consistency.
  • Test prompt engineering effectiveness and response quality.
  • Validate Retrieval-Augmented Generation (RAG) outputs.
  • Measure hallucination rates and response reliability.

Skills

QA/Test Engineering
Generative AI testing
AI Agents / Agentic AI / RAG
Prompt engineering
Python
API testing
Automation frameworks
AI platforms knowledge

Tools

Postman
Swagger
REST APIs
Selenium
Playwright
PyTest

Job description

TALENT Software Services is seeking an AI Agentic Tester for a remote Denver-based role. You will validate AI agent workflows, test end-to-end AI use cases, and ensure output quality across multi-agent interactions.

The ideal candidate has 3–8 years of QA/testing experience, strong Python skills, API testing experience, and familiarity with AI platforms such as Azure OpenAI, AWS Bedrock, or Gemini to ensure robust AI performance.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Engineer (AgentQ)
AI Engineer (AgentQ)

Virtusa Corporation • United States

Remote
USD 120,000 - 180,000
AI Tester at Denvor CO _Remote
AI Tester at Denvor CO _Remote

Volto Consulting • Colorado

Remote
USD 90,000 - 130,000
Senior QA Automation Engineer, AI Agentic Systems (Remote)
Senior QA Automation Engineer, AI Agentic Systems (Remote)

GCS Recruitment • United States

Remote
USD 120,000 - 170,000
Senior QA Engineer - AI & Agentic Systems (Remote)
Senior QA Engineer - AI & Agentic Systems (Remote)

GCS Recruitment • United States

Remote
USD 120,000 - 160,000
Senior/Staff QA Engineer – Agentic AI Platform
Senior/Staff QA Engineer – Agentic AI Platform

Ontrac Solutions LLC. • Chicago (IL)

On-site
USD 110,000 - 124,000
Manual Tester wtih Agentic AI
Manual Tester wtih Agentic AI

Tracker-RMS • Boston (MA)

On-site
USD 120,000 - 160,000
Medical plans
401(k) starting day one
Life insurance
+2
AI Test Lead
AI Test Lead

Envision Technology Solutions • Frisco (TX)

On-site
USD 110,000 - 165,000
Senior QA Engineer — AI Agent Platform
Senior QA Engineer — AI Agent Platform

Ontrac Solutions LLC. • Chicago (IL)

Hybrid
USD 110,000 - 124,000
AI Testing Engineer: Automation & LLM QA
AI Testing Engineer: Automation & LLM QA

Robotic Process Automation LLC • Irving (TX)

On-site
USD 80,000 - 140,000
AI Tester: QA for AI/LLM Apps & APIs (Chicago)
AI Tester: QA for AI/LLM Apps & APIs (Chicago)

Robotic Process Automation LLC • Chicago (IL)

On-site
USD 83,000 - 107,000