AI Agentic Tester

Delan Associates, Inc.; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Delan Associates, Inc. in Denver, MO (Remote) is seeking an AI Agentic Tester to validate AI agent workflows and end-to-end AI applications.

You will assess LLM responses for accuracy, relevance, and hallucinations, test RAG pipelines, and automate API tests using Postman, Swagger, Python, Selenium or Playwright.

Qualifications

• Experience in AI agentic and generative AI testing.
• Strong knowledge of LLM evaluation metrics.
• Hands-on with API testing and automation.
• Proficient in Python and test automation tools.
• Familiar with Azure OpenAI / AWS Bedrock / Gemini.

Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.

Skills

AI Agentic Testing

Functional Testing

Generative AI Testing

Large Language Models (LLMs)

AI Agents & Agentic AI

Retrieval-Augmented Generation (RAG)

Prompt Engineering Validation

AI Model Validation & Evaluation

API Testing

Python

Test Automation (Selenium / Playwright

PyTest

Azure OpenAI / AWS Bedrock / Gemini

End-to-End Workflow Testing

Multi-Agent Testing

Tools

Postman

REST APIs

Swagger

Selenium

Playwright

PyTest

Job description

AI Agentic Tester Denver, MO (Remote)

Must-Have Skills

• AI Agentic Testing
• Functional Testing
• Generative AI Testing
• Large Language Models (LLMs)
• AI Agents & Agentic AI
• Retrieval-Augmented Generation (RAG)
• Prompt Engineering Validation
• AI Model Validation & Evaluation
• API Testing (Postman, REST APIs, Swagger)
• Python
• Test Automation (Selenium / Playwright / PyTest)
• AI Evaluation Metrics (Accuracy, Hallucination, Relevance, Consistency)
• End-to-End Workflow Testing
• Multi-Agent Testing
• Azure OpenAI / AWS Bedrock / Gemini

Core Responsibilities

• Perform functional testing of AI agent workflows and end-to-end AI applications.
• Validate LLM responses for accuracy, relevance, consistency, and hallucination rates.
• Test RAG pipelines, prompt engineering, and AI agent orchestration.
• Execute API testing using Postman, Swagger, and REST APIs.
• Develop and maintain automation scripts using Python, Selenium, Playwright, or PyTest.
• Validate multi-agent interactions, tool calling, and workflow execution.
• Apply AI testing methodologies and evaluation metrics to ensure response quality.
• Work with Azure OpenAI, AWS Bedrock, Gemini, or similar AI platforms to validate AI solutions.

Originally posted on Himalayas