AI Agentic Tester

Delan Associates, Inc; Denver (MO); Remote; USD 70,000 - 110,000; Full time

Job summary

Delan Associates, Inc. is seeking an AI Agentic Tester in Denver, MO (Remote) to validate AI agent workflows and end-to-end AI applications. The role emphasizes testing of LLM responses, tool calls, and RAG pipelines, with an emphasis on accuracy and hallucination metrics.

The ideal candidate will develop automation scripts in Python and leverage Postman, Swagger, Selenium, Playwright, or PyTest to ensure high-quality AI solutions across Azure OpenAI, AWS Bedrock, and Gemini platforms.

Qualifications

• Must-Have Skills listed in the job ad.
• Experience validating AI agent workflows and end-to-end AI applications.
• Proficiency with LLM evaluation, accuracy, hallucination, and relevance metrics.

Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.

Skills

AI Agentic Testing

Functional Testing

Generative AI Testing

LLMs

AI Agents & Agentic AI

RAG

Prompt Engineering Validation

AI Model Validation & Evaluation

API Testing

Python

Test Automation

AI Evaluation Metrics

End-to-End Workflow Testing

Multi-Agent Testing

Azure OpenAI

AWS Bedrock

Gemini

Tools

Postman

Swagger

REST APIs

Selenium

Playwright

PyTest

Python

Job description

AI Agentic Tester

Denver, MO (Remote)

Must-Have Skills

• AI Agentic Testing
• Functional Testing
• Generative AI Testing
• Large Language Models (LLMs)
• AI Agents & Agentic AI
• Retrieval-Augmented Generation (RAG)
• Prompt Engineering Validation
• AI Model Validation & Evaluation
• API Testing (Postman, REST APIs, Swagger)
• Python
• Test Automation (Selenium / Playwright / PyTest)
• AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)
• End-to-End Workflow Testing
• Multi-Agent Testing
• Azure OpenAI / AWS Bedrock / Gemini

Core Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.