Senior Automation QA Engineer, Bangalore

Trintech Solutions Private Limited (India); Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Job summary

Trintech Solutions Private Limited (India) is looking for a QA Automation Engineer with over 5 years of experience in API and web-based application testing. Your role will include designing and implementing test strategies, building test suites, and integrating them into Azure DevOps CI/CD pipelines.

The ideal candidate will have strong skills in Pytest and Playwright, as well as experience in REST API testing. Join a cross-functional team focused on enhancing quality in AI systems.

Qualifications

• 5+ years of QA automation experience for API and web-based applications.
• Experience with Pytest and Playwright is required.
• Strong proficiency in scripting with Python preferred.

Responsibilities

• Design and implement test automation for backend APIs and Angular UIs.
• Build and maintain test suites for Python backend services.
• Integrate test suites into Azure DevOps CI/CD pipelines.

Skills

QA automation experience

Pytest

Playwright

REST API testing

Python scripting

SQL/NoSQL

CI/CD integration

LLM Testing Awareness

Job description

What You’ll Do

Design and implement test automation strategies for backend APIs, agent orchestration logic, and Angular-based agent UIs.

Build and maintain Pytest-based test suites for Python backend services and agent workflow validation.

Implement Playwright automation for Angular-based UI, covering interaction flows, approval queues, and streaming response handling.

Validate multi-step agent workflows by asserting on intermediate states, tool call outputs, and handoff results, not just final API responses.

Use Langfuse traces to validate agent reasoning chains, confidence scores, and execution paths.

Build and maintain LLM evaluation harnesses covering prompt regression testing, RAG output validation, and model version change detection.

Integrate test suites into Azure DevOps CI/CD pipelines as automated quality gates.

Verify audit log integrity, including immutable log structure, correlation ID chains, and SOX‑compliant event emission.

Who You Are

• 5+ years of QA automation experience for API and web‑based applications.
• Pytest as the primary test framework for Python backend and agent logic validation.
• Playwright for UI automation targeting Angular‑based interfaces.
• REST API testing using Postman, HTTPX, or equivalent tools for contract and schema validation.
• Python for test scripting, strongly preferred given the stack.
• SQL/NoSQL for test data setup, teardown, and database state validation.
• Azure DevOps CI/CD pipeline integration for automated quality gates.
• LLM / Agent Testing Awareness – understands that LLM outputs are non‑deterministic and knows why standard assertion patterns must be adapted, even without prior hands‑on production experience.
• Familiarity with multi‑step agent graph testing, including asserting on intermediate states, tool calls, and handoff outputs rather than just end results.
• Understanding of what Langfuse traces contain and how they can be used for agent validation.
• Awareness of model version regression risks and how to detect quality degradation when model versions change.

Nice to Have

• Hands‑on Langfuse experience covering trace validation, eval pipeline setup, and confidence score assertion.
• Experience with LLM evaluation frameworks such as Ragas or equivalent for RAG output quality.
• SOX/GDPR audit trail validation, including verifying immutable log integrity.
• Azure Monitor or Application Insights for observability‑assisted debugging.

What You’ll Learn & Gain

A front‑row seat to defining what quality means for AI agent systems, working on test problems that don’t yet have established industry playbooks. Hands‑on experience building LLM evaluation harnesses and agent validation frameworks in a production financial platform context. Deep familiarity with Langfuse, agent trace analysis, and confidence‑based quality gates – skills increasingly valuable across the AI engineering industry.

Collaboration with a cross‑functional squad of backend, frontend, and AI/ML engineers where QA is treated as a first‑class engineering discipline.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or disability.