Remote AI QA Engineer for LLMs & RAG Pipelines

Jobgether; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Fully remote

International teams

Learning opportunities

Mentorship

Job summary

Jobgether is seeking a Quality Engineer focused on AI/LLM testing to design and optimize validation pipelines for chatbots, RAG systems, and autonomous agents.

You will build automation in Python using Robot Framework, Playwright, and CI/CD integration, collaborating with AI/ML, product, and DevOps teams to raise testing standards.

This fully remote US role offers full-time engagement, competitive benefits, and exposure to cutting-edge AI technologies across cloud platforms.

Qualifications

• Bachelor's degree in Computer Science or related field; equivalent practical experience valued.
• Strong Python skills and ability to build automation utilities.
• Solid understanding of software testing principles and modern automation practices.
• Hands-on experience with AI/LLM testing, prompt engineering, and evaluation frameworks.
• Experience designing automated output evaluation, hallucination detection, and safety validation.
• Experience API testing, microservices, UI automation, and SQL data validation.
• Familiarity with AI agent frameworks (LangChain, LangGraph, CrewAI, Semantic Kernel).
• Experience integrating automated testing into CI/CD pipelines and cloud platforms (AWS, Azure, GCP).
• Strong debugging, analytical, and problem-solving abilities in Agile contexts.

Responsibilities

• Design, implement, and optimize testing strategies for AI-powered applications and RAG pipelines.
• Develop AI evaluation frameworks to assess hallucinations, bias, safety, and response quality.
• Build automated validation pipelines for LLM outputs using structured assertions and metrics.
• Test agentic systems including multi-agent orchestration, memory, and tool calls.
• Contribute to prompt and context engineering to improve AI reliability.
• Evaluate performance, latency, and cost of AI workloads across cloud infra.
• Create scalable UI/API automation frameworks using Python and Robot Framework.
• Automate testing across REST/GraphQL, events, streaming, and microservices.
• Validate RAG architectures and vector databases for retrieval accuracy.

Skills

Python

Automated testing

AI/LLM testing

CI/CD

Debugging

Education

Bachelor's degree in CS or related

Tools

Robot Framework

Playwright

Selenium

LangChain

Job description

Jobgether is seeking a Quality Engineer focused on AI/LLM testing to design and optimize validation pipelines for chatbots, RAG systems, and autonomous agents.

You will build automation in Python using Robot Framework, Playwright, and CI/CD integration, collaborating with AI/ML, product, and DevOps teams to raise testing standards.

This fully remote US role offers full-time engagement, competitive benefits, and exposure to cutting-edge AI technologies across cloud platforms.