Staff Test Engineer/ Senior Staff Test Engineer (AI)

Tally Solutions; Bengaluru; On-site; INR 2,000,000 - 2,800,000; Full time

Job summary

Tally Solutions in Bengaluru is seeking a Staff Test Engineer / Senior Staff Test Engineer to own end-to-end quality for AI/LLM-based products. You will drive test strategy, automation, and risk management across requirements, architecture discussions, release, and production validation.

You will lead QA efforts, validate AI agents and tool calls, and ensure robust testing across API, performance, and non-functional requirements, collaborating with AI and engineering teams.

Qualifications

• Bachelor's degree in Computer Science or related field.
• Proven experience with AI/LLM testing in product environments.
• Strong experience in API testing and test automation.
• Experience leading QA efforts in complex products.

Responsibilities

• Quality Ownership: Own QA activities across the complete project lifecycle from requirements and design to release and production validation.
• Test Strategy Planning: Define test strategy, scope, test coverage, automation approach, environments, test data, and release criteria.
• Testing Execution: Drive functional, API, integration, regression, performance, and NFR testing based on project requirements.
• Risk Management: Proactively identify technical, functional, integration, and release risks, define mitigation plans, and communicate them early.
• Release Readiness: Take ownership of release readiness and provide data-driven quality recommendations.
• Production Support: Analyse production issues, drive root-cause analysis, and implement preventive quality improvements.

Skills

AI/LLM testing

Programming in Python/JavaScript

API testing

Test automation

CI/CD integration

Stakeholder communication

Technical leadership

Education

Bachelor's degree in Computer Science or related

Tools

JMeter

Gatling

k6

CI/CD tooling

Job description

Job Summary

We are looking for a Staff Test Engineer / Senior Staff Test Engineer with strong technical expertise and proven experience in AI/LLM-based projects. The candidate will own quality end-to-end, from requirements and architecture discussions through testing, release, and production validation. The role requires a combination of AI testing expertise, strong programming skills, test strategy, risk management, stakeholder communication, and technical leadership. The candidate should be able to independently drive quality for complex products and influence engineering and product decisions.

Responsibilities

• Quality Ownership: Own QA activities across the complete project lifecyclefrom requirements and design to release and production validation.
• Test Strategy Planning: Define test strategy, scope, test coverage, automation approach, environments, test data, and release criteria.
• Testing Execution: Drive functional, API, integration, regression, performance, and NFR testing based on project requirements.
• Risk Management: Proactively identify technical, functional, integration, and release risks, define mitigation plans, and communicate them early.
• Release Readiness: Take ownership of release readiness and provide data-driven quality recommendations.
• Production Support: Analyse production issues, drive root-cause analysis, and implement preventive quality improvements.

The Scope of Your Influence

Where Your Expertise Meets Our Future

AI / LLM Testing

• Hands-on experience working with AI/ML/LLM-based products or projects.
• Strong understanding of LLMs, Generative AI, Prompt Engineering, and AI-based workflows.
• Experience defining evaluation strategies for AI systems, including accuracy, consistency, hallucination, robustness, context handling, and response quality.
• Experience validating AI agents, tool/function calling, structured outputs, or RAG-based applications is preferred.
• Ability to identify AI-specific edge cases and define appropriate test data, benchmarks, and quality metrics.
• Exposure to LLM evaluation, model comparison, and AI-assisted testing is an advantage.

Technical Quality Engineering

• Strong programming skills in Python, JavaScript, or similar, with the ability to build test automation and utilities.
• Strong understanding of API testing, databases/SQL, system architecture, networking, distributed systems, and debugging.
• Experience designing scalable automation frameworks and integrating automated tests into CI/CD pipelines.
• Exposure to performance/scalability testing using tools such as JMeter, Gatling, or k6.
• Good understanding of cloud platforms, containers, logging, monitoring, and deployment architecture.