Quality Engineering consultant

Artech L.L.C.; Bengaluru; On-site; INR 600,000 - 1,200,000; Full time

Job summary

Artech L.L.C. is seeking a Quality Engineering (QE) Consultant with hands-on experience in AI/GenAI testing, API testing, and Playwright automation. You will design test strategies for AI-powered applications, validate AI model behavior, and drive quality across CI/CD pipelines.

The role collaborates with Product Owners, Developers, Data Scientists, and AI Engineers to ensure reliability, safety, and performance of AI-enabled solutions throughout the software lifecycle.

Qualifications

• 4–6 years of experience in Quality Engineering, Software Testing, or Test Automation.
• Hands-on experience testing AI/Generative AI applications.
• Experience with modern web application testing and API testing.
• Familiarity with Gemini and Agentic AI frameworks.
• Knowledge of Azure DevOps or Jira for test management.

Responsibilities

• Design and execute test strategies for AI/GenAI applications including LLM-powered solutions.
• Develop and maintain Playwright-based test automation frameworks.
• Automate UI, API, and end-to-end business workflows.
• Define AI evaluation metrics and quality scorecards.
• Collaborate with Product Owners, Developers, Data Scientists, and AI Engineers to drive quality.

Skills

AI/GenAI testing

Playwright automation

API testing

CI/CD integration

QA strategy & metrics

Python/TypeScript automation

Tools

Postman

REST Assured

Azure DevOps

Jira

Playwright API Testing

Job description

We are seeking a highly motivated Quality Engineering (QE) Consultant Experinece in software testing, automation, and emerging AI technologies. The ideal candidate will have hands‑on experience testing AI/GenAI applications, APIs, and web applications, along with strong expertise in Playwright automation and AI evaluation frameworks.

This role will focus on ensuring the quality, reliability, safety, and performance of AI-powered solutions by designing comprehensive test strategies, implementing automation frameworks, validating AI model behavior, and establishing evaluation mechanisms for large language model (LLM)-based applications.

The candidate will collaborate closely with Product Owners, Developers, Data Scientists, and AI Engineers to drive quality throughout the software development lifecycle.

AI & GenAI Testing

• Design and execute test strategies for AI/GenAI applications, including LLM-powered solutions, AI agents, copilots, and conversational systems.
• Validate AI outputs for:
• Accuracy
• Relevance
• Groundedness
• Consistency
• Toxicity and safety compliance
• Create and maintain prompt test suites and benchmark datasets.
• Perform functional, regression, performance, and reliability testing for AI-enabled applications.
• Define and execute AI evaluation metrics and quality gates.

Automation & Quality Engineering

• Develop and maintain test automation frameworks using Playwright.
• Automate UI, API, and end-to‑end business workflows.
• Create reusable automation assets and testing accelerators.
• Integrate automated tests into CI/CD pipelines.
• Support shift‑left testing and quality engineering practices.

API Testing

• Design and execute API test cases using tools such as:
• Postman
• REST Assured
• Playwright API Testing
• Validate API contracts, authentication, authorization, error handling, and performance.
• Conduct integration testing across distributed systems and third‑party services.

AI Evaluation & Quality Metrics

• Establish AI evaluation frameworks and quality scorecards.
• Measure model quality using evaluation techniques such as:
• Precision/Recall
• Relevancy scoring
• Semantic similarity
• Groundedness validation
• Analyze AI quality trends and recommend improvements.
• Support Responsible AI and model governance requirements.

Collaboration & Stakeholder Management

• Work closely with engineering, product, and AI teams to identify quality risks early.
• Participate in requirement reviews and design discussions.
• Communicate testing progress, risks, and quality metrics to stakeholders.
• Contribute to QE best practices, standards, and reusable playbooks.

Experience

• 4–6 years of experience in Quality Engineering, Software Testing, or Test Automation.
• Hands‑on experience testing AI/Generative AI applications.
• Experience with modern web application testing and API testing.

Technical Skills

• Hands‑on experience with API testing and automation.
• Experience validating AI/LLM-based applications and AI agents.
• Good understanding of AI evaluation methodologies and testing techniques.
• Familiarity with:
• Gemini
• Agentic AI frameworks
• Knowledge of test management tools such as Azure DevOps (ADO), Jira, or similar platforms.
• Experience working with CI/CD pipelines and DevOps practices.

Preferred Skills

• Exposure to Python or TypeScript for automation and AI testing.
• Experience with AI evaluation frameworks such as:
• RAG evaluation
• Human feedback‑based evaluation
• Knowledge of Responsible AI principles, AI governance, and model risk management.
• Experience testing Retrieval‑Augmented Generation (RAG) systems and AI agents.