Senior Test Analyst

Smartstream; Mumbai; On-site; INR 2,500,000 - 3,500,000; Full time

Job summary

SmartStream seeks a Senior QA Engineer to drive the quality strategy for agentic AI systems, including LLM-powered applications, RAG pipelines, and tool-calling. You will ensure accuracy, safety, reliability, and compliance while partnering with global engineering, product, and AI/ML teams.

You will write automation for unit, API, and end-to-end UI tests; integrate LLMs into workflows; and help monitor, debug, and optimize AI-enabled platforms.

Qualifications

• 5+ years of professional software engineering experience.
• Hands-on work with generative AI and agentic systems.
• Experience working in cross-functional product teams.
• Degree in Computer Science, Engineering, or related field (or equivalent experience).
• Strong problem-solving skills and ability to work in fast-evolving technical domains.

Responsibilities

• Write automation alongside development to test unit level, API and e2e UI.
• Implement backend services supporting agent runtimes, tool execution, and memory layers.
• Integrate LLMs into business workflows and internal systems.
• Contribute to orchestration logic connecting tools, APIs, and external services.
• Write clean, maintainable, well-tested code.
• Participate in architectural discussions.
• Collaborate with AI engineers to productionize experimental features.
• Contribute to monitoring, logging, and performance optimization.

Skills

LLM testing

Python

APIs

Prompting

Cloud

Debugging

Analytical thinking

Cross-functional collaboration

Education

CS degree or related field

Tools

PyTest

Playwright

Postman

MLFlow

W&B

Job description

The Senior QA Engineer will drive the quality strategy for agentic AI systems, including applications powered by LLMs, autonomous agents, RAG pipelines, tool-calling, memory modules, and complex AI decisioning flows. The role ensures that AI-driven capabilities meet strict enterprise-grade standards of accuracy, safety, reliability, and compliance, while collaborating closely with engineering, product, and AI/ML teams globally.

Job Responsibilities

• Write automation alongside development to test unit level, API and e2e UI
• Implement backend services supporting agent runtimes, tool execution, and memory layers
• Integrate LLMs into business workflows and internal systems
• Contribute to orchestration logic connecting tools, APIs, and external services
• Write clean, maintainable, well-tested code
• Participate in architectural discussions
• Collaborate with AI engineers to productionize experimental features
• Contribute to monitoring, logging, and performance optimization

Requirements

• Strong hands-on experience testing LLM-based or agentic AI applications
• Expertise in Python automation (PyTest, Playwright, Postman)
• Understanding of prompts, embeddings, vector stores, tokenization
• Experience testing RAG pipelines and grounding data workflows
• Proficiency with APIs, microservices, and cloud environments (Azure/AWS/GCP)
• Ability to evaluate unstructured AI output using structured scoring techniques
• Strong analytical and debugging skills in complex AI-driven workflows

Desirable Skills

• Experience with AI evaluation tools: TruLens, PromptLayer, MLFlow, W&B.
• Familiarity with agentic frameworks: LangChain, AutoGen, Semantic Kernel, CrewAI
• Understanding of LLM safety concepts, adversarial testing, guardrails, red teaming
• Knowledge of model fine-tuning, RLHF, and observability pipelines
• Exposure to distributed architecture and enterprise platforms
• Ability to mentor QA engineers on AI testing methods

Qualifications

• 5+ years of professional software engineering experience
• Hands-on work with generative AI and agentic systems
• Experience working in cross-functional product teams
• Degree in Computer Science, Engineering, or related field (or equivalent experience)
• Strong problem-solving skills and ability to work in fast-evolving technical domains
• Interest in AI-native software architectures and continuous learning mindset
• Certifications such as ISTQB Advanced, Azure/AWS/Google AI certifications (optional)

Equality Statement

SmartStream is an equal opportunities employer. We are committed to promoting equality of opportunity and following practices which are free from unfair and unlawful discrimination.