AI Agentic Tester

Delan Associates, Inc; Denver (CO); Hybrid; USD 90,000 - 120,000; Full time

Job summary

Delan Associates, Inc. seeks an AI Agentic Tester to validate end-to-end AI workflows across agentic systems.

You will perform functional testing of AI agentic workflows and assess LLM outputs for accuracy, relevance, and hallucination rates while coordinating RAG pipelines and prompt engineering validation. In a remote-friendly role based near Denver, you will develop and maintain automated tests using Python, Selenium, Playwright or PyTest, and execute API tests with Postman and Swagger

Qualifications

• Proven experience in AI agentic testing and functional testing.
• Experience validating LLM responses for accuracy, relevance and hallucinations.
• Ability to test RAG pipelines, prompt engineering and agent orchestration.

Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.

Skills

AI Agentic Testing

Functional Testing

Generative AI Testing

Large Language Models

AI Agents

RAG

Prompt Engineering Validation

AI Model Validation

API Testing

Python

Test Automation

AI Evaluation Metrics

End-to-End Workflow Testing

Multi-Agent Testing

Azure OpenAI

AWS Bedrock

Gemini

Tools

Selenium

Playwright

PyTest

Postman

Swagger

Azure OpenAI

AWS Bedrock

Gemini

Job description

AI Agentic Tester

Denver, MO (Remote)

Must-Have Skills

• AI Agentic Testing
• Functional Testing
• Generative AI Testing
• Large Language Models (LLMs)
• AI Agents & Agentic AI
• Retrieval-Augmented Generation (RAG)
• Prompt Engineering Validation
• AI Model Validation & Evaluation
• API Testing (Postman, REST APIs, Swagger)
• Python
• Test Automation (Selenium / Playwright / PyTest)
• AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)
• End-to-End Workflow Testing
• Multi-Agent Testing
• Azure OpenAI / AWS Bedrock / Gemini

Core Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.