Quality Engineer

The Premier Consultants ( Recruitment Company ); Ahmedabad District; On-site; INR 900,000 - 1,500,000; Full time

Job summary

The Premier Consultants invites applications for an AI QA Engineer position based in Ahmedabad. We seek a candidate with 3+ years of QA experience in AI-enabled applications to own quality across AI-driven dashboards, CRMs, and RFP platforms.

This role blends AI, web applications, and quality engineering to deliver reliable AI solutions for global clients. You will design test strategies for LLM-powered features, build evaluation datasets, and validate AI outputs for accuracy and safety.

Qualifications

• Must have QA experience in AI-enabled apps or LLM features.
• Experience with Generative AI, LLM testing, and prompt engineering.
• Familiarity with testing frameworks and API testing.
• Knowledge of SQL, databases, and cloud platforms is a plus.

Responsibilities

• Design and execute test strategies for LLM-powered features and AI agents.
• Build evaluation suites, golden datasets, and prompt regression tests.
• Validate AI responses for accuracy, hallucinations, and formatting.
• Test agentic workflows, tool calling, and multi-step reasoning.
• Perform adversarial testing including prompt injection and edge cases.
• Own end-to-end QA for AI-powered web apps and dashboards.
• Test REST APIs, webhooks, and third-party integrations.
• Ensure security, multi-tenant integrity, and authentication controls.
• Integrate automated testing into CI/CD pipelines.
• Monitor production AI quality, latency, and cost.

Job description

We're Hiring: AI QA Engineer (Generative AI & Web Applications)

Experience: 3+ Years

Working Days: 5 Days a Week

Are you passionate about ensuring the quality, reliability, and performance of AI-powered applications?

We're looking for an experienced AI QA Engineer to own quality across a growing portfolio of AI-driven products, including AI-powered dashboards, CRMs, RFP platforms, lead generation systems, agentic workflows, and workflow automations. This is an exciting opportunity to work at the intersection of Artificial Intelligence, Web Applications, and Quality Engineering, ensuring production-ready AI solutions for global clients.

Key Responsibilities

AI & LLM Testing

• Design and execute test strategies for LLM-powered features, AI agents, RAG systems, and workflow automations.
• Build and maintain evaluation suites, golden datasets, and prompt regression tests.
• Validate AI responses for factual accuracy, hallucinations, formatting, instruction-following, and overall quality.
• Test agentic workflows, tool calling, multi-step reasoning, error handling, latency, and cost optimization.
• Perform adversarial testing, including prompt injection, jailbreak attempts, and edge-case validation.

Application Quality Assurance

• Own end-to-end QA for AI-powered web applications, dashboards, CRMs, and client portals.
• Test REST APIs, webhooks, and third-party integrations.
• Validate database integrity, multi-tenant security, authentication, authorization, and role-based access.
• Conduct performance, stability, and file-processing validation.

Automation & Quality Engineering

• Integrate automated testing into CI/CD pipelines.
• Monitor production systems for AI quality, latency, token usage, and performance.
• Collaborate closely with AI Engineers, Full Stack Developers, Product Teams, and Delivery Managers.
• Document defects, maintain QA dashboards, and contribute to AI QA best practices.

Required Skills

• 3+ years of QA experience, including testing AI-enabled applications or LLM-powered features.
• Experience with Generative AI, LLM testing, Prompt Engineering, and Prompt Regression Testing.
• Hands-on experience with Promptfoo, DeepEval, RAGAs, LangSmith, Braintrust, or similar evaluation frameworks.
• Experience with Playwright, Cypress, or Selenium.
• API testing using Postman or similar tools.
• Good understanding of SQL, databases, CI/CD, Git, and cloud platforms (AWS, Azure, or GCP).
• Knowledge of vector databases, embeddings, and Retrieval-Augmented Generation (RAG).

Preferred Skills

• Experience with n8n, Make, Zapier, or AI agent frameworks.
• Familiarity with Langfuse, Helicone, LangSmith, or AI observability platforms.
• Experience with OpenAI, Claude, or other production LLM APIs.
• Knowledge of Supabase, PostgreSQL, MCP, or client-facing SaaS platforms.
• Exposure to prompt injection testing, red teaming, SOC 2, or compliance testing.

What We're Looking For

• Strong analytical and problem-solving skills
• Passion for AI quality and automation
• Excellent debugging and communication skills
• Ability to work collaboratively in a fast-paced environment
• Proactive mindset with attention to detail
• If you're excited about building reliable, high-quality AI products and shaping the future of AI Quality Assurance, we'd love to hear from you!