AI Agentic Tester

Delan Associates, Inc; St. Louis (MO); Hybrid; USD 90,000 - 130,000; Full time

Job summary

Delan Associates, Inc is seeking an AI Agentic Tester to work remotely from Denver, MO. The role focuses on validating AI agentic workflows, ensuring accuracy and timely responses from LLMs, and testing RAG pipelines and tool integrations.

Proficiency in Python-based automation and API testing is essential. The ideal candidate will execute automated tests using Selenium, Playwright, or PyTest, and collaborate with cloud AI platforms like Azure OpenAI, AWS Bedrock, or Gemini to validate AI

Qualifications

• Must have hands-on experience with AI agentic testing and validating AI workflows.
• Proficiency in functional and end-to-end testing across AI-enabled applications.
• Strong ability to test and validate LLM outputs, including accuracy, relevance and hallucination rates.

Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.

Skills

AI Agentic Testing

Functional Testing

Generative AI Testing

LLMs

Agentic AI

RAG

Prompt Validation

AI Model Evaluation

API Testing

Python

Test Automation

AI Metrics

End-to-End Testing

Multi-Agent Testing

AI Platforms

Tools

Selenium

Playwright

PyTest

Postman

Swagger

Azure OpenAI

AWS Bedrock

Gemini

Job description

AI Agentic Tester

Denver, MO (Remote)

Must-Have Skills

• AI Agentic Testing
• Functional Testing
• Generative AI Testing
• Large Language Models (LLMs)
• AI Agents & Agentic AI
• Retrieval-Augmented Generation (RAG)
• Prompt Engineering Validation
• AI Model Validation & Evaluation
• API Testing (Postman, REST APIs, Swagger)
• Python
• Test Automation (Selenium / Playwright / PyTest)
• AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)
• End-to-End Workflow Testing
• Multi-Agent Testing
• Azure OpenAI / AWS Bedrock / Gemini

Core Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.