Quality Assurance Manager

Tekion Corp; Bengaluru; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

Tekion Corp in Bengaluru seeks an experienced QA Manager to lead the quality engineering function with a focus on test automation and AI/LLM evaluation. You will manage and mentor QA/automation teams and own the automation strategy and quality roadmap for the product.

You will design scalable automation using Playwright, drive AI/LLM evaluation strategies, and collaborate with Data Science on benchmarks, hallucination checks, and prompt regression testing.

Qualifications

• Total Experience: 10–17 years.
• Team Management Experience: Minimum 2 years (leading QA/automation teams).
• Strong hands-on with Playwright for automated UI/end-to-end testing.
• Solid Java programming skills for building and maintaining test frameworks.
• Experience testing AI/LLM-based systems — prompt testing, model output validation.
• CI/CD integration for automated test suites (Jenkins, GitHub Actions, or similar).
• Familiarity with QA methodologies: functional, regression, integration, performance, exploratory testing.

Responsibilities

• Lead, mentor, and grow a team of QA/automation engineers.
• Define and own the test automation strategy, framework, and quality roadmap.
• Design, build, and maintain scalable automation frameworks using Playwright.
• Drive evaluation strategies for AI/LLM-based features with test harnesses and datasets.
• Collaborate with data science/ML teams to define quality benchmarks for AI systems.
• Write and review robust automation code in Java and integrate with CI/CD.
• Establish QA processes, metrics, and ROI across the team.
• Partner with Product, Engineering, and Data Science to define acceptance criteria and gates.
• Own test planning, risk assessment, release sign-off, and leadership reporting.
• Stay current with AI/LLM testing methods and tooling.

Skills

QA management

AI/LLM testing

Eval skills

LLM concepts

CI/CD integration

QA methodologies

Communication skills

Education

Bachelor's/Master's in CS/Engineering

Tools

Playwright

Selenium

Cypress

Postman

REST Assured

Jenkins

GitHub Actions

Job description

We are looking for an experienced QA Manager to lead our quality engineering function, with a strong focus on test automation, evaluation of AI/LLM-based systems, and team leadership. The ideal candidate combines deep hands-on technical expertise with proven experience managing and mentoring QA teams.

• Total Experience: 10 to 17 years
• Team Management Experience: Minimum 2 years (leading QA/automation teams)
• Employment Type: Full-time
• Lead, mentor, and grow a team of QA/automation engineers, driving their technical and professional development.
• Define and own the overall test automation strategy, framework architecture, and quality roadmap for the product.
• Design, build, and maintain scalable automation frameworks using Playwright for web application testing.
• Drive evaluation (eval) strategies for AI/LLM-based features — designing test harnesses, golden datasets, scoring rubrics, and regression suites for model outputs.
• Partner with data science/ML engineering teams to define quality benchmarks for LLM-powered and AI-driven systems, including hallucination checks, output consistency, and prompt regression testing.
• Write and review robust, maintainable automation code in Java, integrating with CI/CD pipelines.
• Establish QA processes, best practices, and metrics (coverage, defect leakage, automation ROI) across the team.
• Collaborate closely with Product, Engineering, and Data Science stakeholders to define acceptance criteria and quality gates, including for AI-driven features.
• Own test planning, risk assessment, release sign-off, and reporting to leadership.
• Stay current with emerging tools and techniques in AI/LLM testing and evaluation methodologies.

Required Skills & Experience

• 10–17 years of total QA/testing experience, including automation-focused roles.
• Minimum 2 years of experience managing/leading a QA or automation team.
• Strong hands-on experience with Playwright for automated UI/end-to-end testing.
• Solid Java programming skills for building and maintaining test frameworks.
• Demonstrated experience testing AI/LLM-based systems — prompt testing, model output validation, regression testing for generative AI features.
• Practical knowledge of eval skills — designing evaluation frameworks, benchmarks, scoring criteria, and golden datasets for AI systems.
• Working knowledge of LLM concepts (prompting, context windows, hallucinations, RAG, fine-tuning basics) sufficient to design meaningful test/eval strategies.
• Experience with CI/CD integration for automated test suites (Jenkins, GitHub Actions, or similar).
• Strong understanding of QA methodologies: functional, regression, integration, performance, and exploratory testing.
• Excellent communication and stakeholder management skills.

Good to Have

• Experience with additional automation tools (Selenium, Cypress, etc.).
• Exposure to API testing tools (Postman, REST Assured).
• Familiarity with cloud platforms (AWS/Azure/GCP) and containerized test environments.
• Prior experience in a product-based or SaaS company testing AI-powered features.

Qualifications

• Bachelor's/Master's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).