AI QA Engineer

Huptech Hr Solutions; Ahmedabad District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Huptech Hr Solutions in Ahmedabad seeks an AI QA Engineer to own quality across AI-powered products including dashboards and CRMs. The role spans AI layer testing (prompts, agents, RAG) and the full web stack (UI, APIs, databases, integrations) in a multi-tenant context.

You will design test strategies, build golden datasets, and automate tests in Python/JavaScript, collaborating with AI engineers and delivery leads to ensure client-ready, reliable software.

Qualifications

• Bachelor's degree in CS/IT/Engineering or related field.
• 3+ years QA experience for web apps and AI-enabled products, including 1+ year with LLM features.
• Strong Python skills for evaluation harnesses and automation; JavaScript/TypeScript for web testing.
• Hands-on with at least one LLM evaluation framework: Promptfoo, DeepEval, RAGAs, LangSmith, Braintrust, or similar.
• Web app testing with Playwright, Cypress, or Selenium; API testing with Postman or REST Assured.
• Solid understanding of prompt engineering, prompt regression testing, and golden-dataset design.
• SQL and basic schema reasoning to verify data flows and multi-tenancy integrity.
• Familiarity with LLM observability tools (Langfuse, Helicone, LangSmith) and token/cost/latency monitoring.
• Git, CI/CD, and cloud basics (AWS, GCP, or Azure); working knowledge of vector databases and RAG.

Responsibilities

• AI & LLM Quality: design test strategies for LL-powered features, agents, RAG systems, and workflows.
• Build golden datasets, evaluation suites, and prompt regression tests to prevent production breaks.
• Validate output quality: factual accuracy, format, tone, instruction-following, and refusal behavior; test multi-step workflows.
• Stress-test RAG pipelines: retrieval accuracy, chunking, context relevance, citations.
• Own end-to-end QA for custom dashboards, AI CRMs, client portals, and internal tools.
• Functional, regression, cross-browser, and responsive testing of web UIs; API testing for REST endpoints and integrations.
• Database and data-integrity testing for multi-tenant architectures; auth and access-control testing.
• Performance, load, and stability testing; monitor production drift, errors, and cost/token metrics.
• Collaborate with AI engineers and delivery leads; automate testing in Python/JS and integrate into CI/CD; document findings.

Skills

Python

JavaScript/TypeScript

LLM testing

Prompt engineering

Automation

CI/CD

Multi-tenant testing

SQL

Cloud basics

Observability tooling

Education

Bachelor's degree in CS/IT/Engineering

Tools

Playwright

Cypress

Selenium

Postman

REST Assured

Langfuse

Helicone

LangSmith

n8n

MCP servers

Supabase

Postgres

Claude/OpenAI API

Job description

• We''re looking for an AI QA Engineer to own quality across our growing portfolio of AI-powered products custom dashboards, AI CRMs, RFP platforms, lead-gen systems, agentic workflows, and automations delivered to agency clients across the US, UK, and AU.
• This role spans both the AI layer (prompts, agents, RAG, model outputs) and the full application stack around it (web app UI, APIs, databases, integrations, multi-tenant access).
• You''ll catch issues that traditional QA misses and the ones that pure ML evaluation misses the messy middle where a working model still ships a broken product.

Key Responsibilities

• AI & LLM Quality Design test strategies for LLM-powered features, agents, RAG systems, and workflow automations (n8n, custom pipelines, MCP-based systems)
• Build and maintain golden datasets, evaluation suites, and prompt regression tests so model or prompt changes don''t silently break production
• Validate output quality across factual accuracy, format adherence, tone, hallucination rate, instruction-following, and refusal behavior Test agentic and multi-step workflows end-to-end tool calls, handoffs, error recovery, loop detection, cost and latency under load
• Stress-test RAG pipelines: retrieval accuracy, chunking quality, context relevance, citation correctness
• Run adversarial testing prompt injection, jailbreak attempts, malformed inputs, edge-case client data Custom App & Platform Quality Own end-to-end QA for custom-built applications: dashboards, AI CRMs, client portals, and internal tools
• Functional, regression, cross-browser, and responsive testing across web UIs (Next.js, React, similar stacks) API testing for REST endpoints, webhooks, and third-party integrations (CRMs, email, calendars, payment, file storage)
• Database and data-integrity testing especially for multi-tenant architectures where one client''s data must never leak to another Auth and access-control testing: SSO, OAuth, role-based permissions, session handling
• Performance, load, and stability testing on production-bound features File upload, processing, and storage flows a recurring failure point in AI products Cross-Cutting
• Monitor production systems for drift, degradation, errors, and cost/token anomalies; set up observability and alerting
• Collaborate with AI engineers, full-stack developers, prompt designers, and delivery leads to catch issues before they hit clients Automate testing in Python and/or JavaScript; integrate into CI/CD pipelines Document findings, build QA dashboards, and contribute to E2M''s internal AI QA playbook

Required Skills & Qualifications

• Bachelor''s degree in Computer Science, IT, Engineering, or related field
• 3+ years QA experience covering both web applications and AI-enabled products, with at least 1 year hands-on testing LLM-powered features, AI workflows, or agent systems
• Strong Python skills for evaluation harnesses and automation; working JavaScript/TypeScript familiarity for web app testing
• Hands-on experience with at least one LLM evaluation framework: Promptfoo, DeepEval, RAGAs, LangSmith, Braintrust, or similar Web app testing experience with Playwright, Cypress, or Selenium API testing with Postman, REST Assured, or equivalent
• Solid understanding of prompt engineering, prompt regression testing, and golden-dataset design Database fundamentals (SQL, basic schema reasoning) enough to verify data flows and spot multi-tenancy bugs
• Familiarity with LLM observability tools (Langfuse, Helicone, LangSmith) and token/cost/latency monitoring Git, CI/CD principles, and at least one cloud platform (AWS, GCP, or Azure) Working knowledge of vector databases, embeddings, and how RAG systems fail

Nice to Have

• Experience testing multi-tenant SaaS platforms or client-facing custom apps Workflow automation platforms (n8n, Make, Zapier) or agent frameworks
• Exposure to MCP (Model Context Protocol) servers or tool-calling architectures
• Familiarity with Supabase, Postgres, or similar backends commonly used in our stack
• Experience with Claude, OpenAI, or open-source LLM APIs in production Agency or client-services background you understand what "client-ready" means Prompt injection / red-teaming experience SOC 2 or compliance-testing exposure

Soft Skills

• Sharp critical eye you assume AI output (and the app around it) is broken until you''ve verified it
• Comfortable working in ambiguity; AI QA doesn''t have settled playbooks yet Clear written communication bug reports for AI failures need precise reproduction steps
• Collaborative across AI engineers, full-stack devs, delivery managers, and account leads Proactive about catching issues before they reach the client demo.