AdvaRisk - QA Lead - Automation Testing

AdvaRisk; Pune; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

AdvaRisk is seeking a senior QA lead to own end-to-end test automation strategy across UI, API, and backend systems in Pune. You will mentor a QA team of 3+ members, defining requirements with product and data science teams and ensuring robust test coverage and release quality outcomes.

You will build and maintain test plans, triage defects, and drive performance and AI/ML integration testing. Strong communication and CI/CD collaboration are essential in this role.

Qualifications

• Strong QA fundamentals across UI, API, backend and end-to-end testing.
• Hands-on automation experience using Python; able to design, build and scale frameworks.
• People leadership experience with 3+ QA engineers.
• Strong API testing and debugging skills; REST, authentication and error handling.
• Experience with performance testing tools and interpreting results.
• Familiarity with AI/ML evaluation and integration testing.
• Experience validating data and service behavior with logs and metrics.
• Knowledge of frontend tests (JS, HTML, CSS) is advantageous.

Responsibilities

• Own end-to-end test automation strategy and standards across UI, API, backend and integration testing.
• Lead, mentor and manage QA team; drive delivery, prioritization and quality outcomes.
• Clarify requirements with product/data science teams and define acceptance criteria.
• Create and maintain test plans, scenarios, data, traceability and release reports.
• Triages defects and coordinates prioritization with relevant teams.
• Design and execute AI/ML model integration tests and evaluate outputs.
• Validate LLM-enabled workflows for groundedness and prompt sensitivity.
• Perform end-to-end and system integration testing across services and databases.
• Plan and run performance, load and endurance tests; identify bottlenecks.
• Integrate automated checks into CI/CD pipelines.

Skills

QA fundamentals

Python automation

Team leadership

API testing

Performance testing

AI/ML testing

CI/CD

LLM evaluation

SQL & logs analysis

Clear communication

Tools

Pytest

Robot Framework

Selenium

Cypress

JMeter

Locust

k6

SQL

CI/CD pipelines

Job description

• Own and drive the end-to-end test automation strategy, defining the roadmap, frameworks, tooling and standards for UI, API, backend and integration testing.
• Lead, mentor and manage the QA team, owning delivery, prioritization, test coverage and quality outcomes.
• Work closely with engineering, product, data science/ML and client service teams to clarify requirements and define acceptance criteria.
• Create and maintain test plans, test scenarios, test data, traceability and release-level quality reports.
• Triage defects, assess business and customer impact, and coordinate defect prioritization with relevant teams.
• Test AI/ML model integrations end to end, including request/response contracts, preprocessing and post-processing, timeout/retry behavior, failure handling and fallback paths.
• Build model-focused test suites using representative and edge-case datasets; assess output correctness against agreed reference labels or acceptance criteria, and track metrics such as precision/recall where applicable.
• Validate LLM-enabled workflows for groundedness, consistency, hallucination risks, prompt/input sensitivity and regression when prompts, models or versions change; use defined evaluation criteria rather than subjective checks alone.
• Check model-service latency, throughput, rate limits, availability, version compatibility and behavior under malformed, incomplete or unexpected inputs.
• Perform end-to-end, system integration and regression testing across UI, backend services, APIs, databases and third party/model services.
• Establish performance baselines and report findings with actionable recommendations for capacity and scalability improvements.
• Plan and execute performance, load, stress, spike and endurance tests; analyze throughput, latency, concurrency, resource utilization and bottlenecks.
• Work with developers to reproduce issues, verify fixes and prevent regressions throughout the release cycle.
• Own test-suite execution in non-production environments and integrate automated checks into CI/CD pipelines where appropriate.

Key Roles & Responsibilities

• Own and drive the end-to-end test automation strategy, defining the roadmap, frameworks, tooling and standards for UI, API, backend and integration testing.
• Lead, mentor and manage the QA team, owning delivery, prioritization, test coverage and quality outcomes.
• Work closely with engineering, product, data science/ML and client service teams to clarify requirements and define acceptance criteria.
• Create and maintain test plans, test scenarios, test data, traceability and release-level quality reports.
• Triage defects, assess business and customer impact, and coordinate defect prioritization with relevant teams.
• Test AI/ML model integrations end to end, including request/response contracts, preprocessing and post-processing, timeout/retry behavior, failure handling and fallback paths.
• Build model-focused test suites using representative and edge-case datasets; assess output correctness against agreed reference labels or acceptance criteria, and track metrics such as precision/recall where applicable.
• Validate LLM-enabled workflows for groundedness, consistency, hallucination risks, prompt/input sensitivity and regression when prompts, models or versions change; use defined evaluation criteria rather than subjective checks alone.
• Check model-service latency, throughput, rate limits, availability, version compatibility and behavior under malformed, incomplete or unexpected inputs.
• Perform end-to-end, system integration and regression testing across UI, backend services, APIs, databases and third party/model services.
• Establish performance baselines and report findings with actionable recommendations for capacity and scalability improvements.
• Plan and execute performance, load, stress, spike and endurance tests; analyze throughput, latency, concurrency, resource utilization and bottlenecks.
• Work with developers to reproduce issues, verify fixes and prevent regressions throughout the release cycle.
• Own test-suite execution in non-production environments and integrate automated checks into CI/CD pipelines where appropriate.

Required Skills

• Strong experience testing web applications across UI, backend services, APIs, databases and end-to-end workflows.
• Strong QA fundamentals with substantial hands‑on automation experience using Python; able to design, build, maintain and scale automation frameworks rather than only execute existing scripts.
• People leadership experience, including leading, mentoring and driving the output of a team of 3 or more QA engineers.
• Strong API testing and debugging skills, including REST APIs, response/schema validation, authentication, error handling and service-to-service integration.
• Experience with performance‑testing tools such as JMeter, Locust, k6 or equivalent, and ability to interpret results and identify bottlenecks.
• Working knowledge of AI/ML model evaluation and integration testing; able to define test datasets, expected outcomes, thresholds and regression checks in collaboration with data science/ML engineers.
• Familiarity with testing LLM‑based features, including prompt/version regression, groundedness and response‑quality evaluation, structured‑output validation and safe failure handling.
• Comfortable validating data and service behavior using SQL, logs, metrics and traces; familiarity with CI/CD‑based automated test execution is an advantage.
• Good understanding of front‑end technologies and browser‑based testing, including JavaScript, HTML and CSS.
• Hands‑on experience building and maintaining automation frameworks using tools such as Pytest, Robot Framework, Selenium, Cypress or equivalent; experience with API automation is essential.
• Experience working in Agile/Scrum teams and participating in sprint planning, refinement, reviews and release readiness.
• A strong quality mindset and a track record of promoting automation‑first, risk‑based testing and continuous quality improvement.
• Clear communication skills and the ability to work across engineering, product, QA and AI/ML teams to define and deliver quality outcomes.

Behavioral Expectations

• Be excited by and have a positive outlook to navigate ambiguity.
• Passion for results and excellence.
• Team player.
• Must be able to get the job done by working collaboratively with others.
• Inquisitive and analytical mind; out‑of‑the‑box thinking.
• Prioritize among competing opportunities, balancing consumer needs with business and product priorities, and clearly articulate the rationale behind product decisions.
• Straightforward and professional.
• Good communicator.
• Maintain high energy and motivate the team.
• A do-it-self-orientation, consistent with the company's \"roll-up-the-sleeves\" culture.
• Proactive.

(ref:hirist.tech)