Remote AI Agent Quality Engineer (Junior)

CodeRoad; Town of Montana (WI); On-site; USD 65,000 - 90,000; Full time

Job summary

CodeRoad is seeking a Junior AI QA Engineer to help validate the next generation of autonomous systems. You will evaluate Agentic AI and RAG, ensuring reliable task planning, correct tool usage, and error recovery without hallucinations.

This role emphasizes Python-based automation, API testing, and rigorous input/output validation. Join a fully remote team in a fast-growing company delivering nearshore software solutions.

Qualifications

• Strong knowledge of Agentic workflows and prompt engineering.
• English proficiency for global collaboration.
• Proficiency in Python 3.10+ for automation.
• Experience testing RESTful APIs and JSON structures.

Responsibilities

• Create question templates and Python scripts to test AI Agents and RAG instances.
• Evaluate Agent performance using KPI metrics: Success Rate, Tool Use Accuracy, Planning Quality, and Autonomy.
• Collaborate with AI Engineers to build Ground Truth datasets and Agentic Task Datasets.
• Write unit and integration tests for Python-based RESTful APIs and agent endpoints.
• Run load and bulk testing with Locust or JMeter to assess high-volume handling.
• Test input/output payloads to catch prompt injection risks and logic failures.

Skills

Agentic AI

LLM orchestration

Python scripting

REST API testing

Analytical thinking

Advanced English

Education

Bachelor's in Computer Science

Tools

Pytest

Playwright

Selenium

LangSmith

Langfuse

OpenTelemetry

Job description

CodeRoad is seeking a Junior AI QA Engineer to help validate the next generation of autonomous systems. You will evaluate Agentic AI and RAG, ensuring reliable task planning, correct tool usage, and error recovery without hallucinations.

This role emphasizes Python-based automation, API testing, and rigorous input/output validation. Join a fully remote team in a fast-growing company delivering nearshore software solutions.