AI QA Engineer: Prompt & RAG Testing for AI Apps

Eurisko; Riyadh; On-site; SAR 180,000 - 240,000; Full time

Job summary

Eurisko is seeking an AI QA Engineer to test and validate applications powered by large language models and generative AI. The role blends traditional QA with prompt testing, AI response evaluation, RAG validation, and automated quality assessments.

You will collaborate with AI engineers, developers, and product teams to identify failure patterns, set quality standards, and prevent regressions across AI-powered applications.

Qualifications

• Professional experience in software QA, including test planning, test-case design, defect management, API testing, and test automation.
• Hands-on experience testing LLM-powered or generative AI applications, with a strong understanding of prompt behavior and AI-specific failure modes.
• Working knowledge of RAG, embeddings, vector stores, context management, and tool calling.
• Practical Python skills for test automation, dataset preparation, API integration, and evaluation scripts.
• Ability to define measurable evaluation criteria and test systems whose outputs may vary between runs, rather than relying solely on exact-text assertions.
• Experience working in Agile/Scrum teams, with strong analytical, documentation, and communication skills.

Responsibilities

• Test prompts and conversational behavior: Design and execute test scenarios covering system instructions, prompt variations, ambiguous inputs, edge cases, multilingual interactions, and multi-turn conversations.
• Evaluate AI-generated responses: Assess accuracy, relevance, completeness, tone, and adherence to business requirements; identify hallucinations and unsafe outputs.
• Validate RAG applications: Test retrieval quality, grounding, citation accuracy, and behavior when information is missing or outdated.
• Build automated evaluation and regression suites: Create representative test datasets, scoring criteria, and repeatable evaluations.
• Perform safety and adversarial testing: Test prompt injection, data exposure risks, and guardrails adherence.
• Test end-to-end AI workflows and report quality: Validate APIs, agent actions, tool calls, memory, latency, and integration behavior.

Skills

Software QA

LLM testing

Python

Agile testing

Job description

Eurisko is seeking an AI QA Engineer to test and validate applications powered by large language models and generative AI. The role blends traditional QA with prompt testing, AI response evaluation, RAG validation, and automated quality assessments.

You will collaborate with AI engineers, developers, and product teams to identify failure patterns, set quality standards, and prevent regressions across AI-powered applications.