QA Agentic AI

Sycamore Informatics Inc.; India; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Sycamore Informatics Inc. is seeking an Agentic AI QA Analyst to validate quality, reliability, safety, and correctness of Agentic AI systems. You will test autonomous agents, tool calls, and memory handling, ensuring robust prompt chaining, non-deterministic outputs, and escalation processes.

The role emphasizes planning and reasoning tests, log inspection, and cross-team coordination in regulated environments where applicable. 2–5 years of relevant experience is expected.

Qualifications

• Bachelor's degree in a related field or equivalent experience.
• Experience testing autonomous/multi-agent AI systems.
• Strong analytical skills for non-deterministic AI behavior.
• Proficiency with Python-based automation and API testing.
• Familiarity with testing toolchains and log analysis.

Responsibilities

• Test agentic AI workflows involving planning, reasoning, action, and reflection loops.
• Validate autonomous decision-making paths and goal completion behavior.
• Test prompt chaining, context persistence, and memory handling.
• Validate tool calling (APIs, databases, search, RPA, internal services).
• Design tests for hallucinations, partial failures, retries, and recovery logic.
• Validate human-in-the-loop checkpoints, approvals, overrides, and escalation paths.
• Perform functional, exploratory, regression, and scenario-based testing.
• Document defects with reproducible prompts, agent state, tools invoked, and outputs.
• Support risk-based testing and compliance requirements (CSA / GxP where applicable).
• Coordinate with cross-functional teams to deliver quality results.

Skills

Agentic AI testing

Multi-agent testing

Prompt testing

LLM tool calling

API testing

Logs & traces

RAG knowledge

Education

Bachelor's degree in CS/EE/Physics/Math or related

Tools

Python

Robot Framework

REST/GraphQL

Job description

Experience (in years)

- 2 - 5 years (total experience - expected for the role)

Education

- e.g., Bachelor’s degree, preferably in Computer Science, Electrical Engineering, Physics, Math or any other related discipline.

Summary

The Agentic AI QA Analyst is responsible for validating the quality, reliability, safety, and correctness of Agentic AI systems. This includes testing autonomous agents, multi-step reasoning flows, tool orchestration, and human-in-the-loop controls. The role requires strong analytical skills to evaluate non-deterministic AI behavior.

• Test agentic AI workflows involving planning, reasoning, action, and reflection loops
• Validate autonomous decision-making paths and goal completion behavior
• Test prompt chaining, context persistence, and memory handling
• Validate tool calling (APIs, databases, search, RPA, internal services)
• Design tests for hallucinations, partial failures, retries, and recovery logic
• Validate human-in-the-loop checkpoints, approvals, overrides, and escalation paths
• Perform functional, exploratory, regression, and scenario-based testing
• Document defects with reproducible prompts, agent state, tools invoked, and outputs
• Support risk-based testing and compliance requirements (CSA / GxP where applicable)
• Under the Sycamore Product and coordinate with different teams to get the work done

Technical Skills – Agentic AI Testing

• Understanding of Agentic AI architectures (Planner–Executor, ReAct, AutoGPT-style agents)
• Experience testing multi-agent systems and agent-to-agent communication
• Prompt testing and prompt-chain validation techniques
• Testing non-deterministic outputs using evaluation heuristics and acceptance bands
• Experience validating LLM tool calling and function calling
• API testing (REST/GraphQL) for agent-integrated services
• Ability to inspect logs, traces, and agent execution graphs
• Familiarity with vector databases, embeddings, and retrieval-augmented generation (RAG)
• Understanding of model limitations, hallucination patterns, and AI failure modes

Deep-Dive Testing Experience (Agentic AI)

• Testing planning accuracy and goal decomposition across multi-step tasks
• Validating agent memory (short-term vs long-term) and context carryover
• Testing tool misuse, incorrect tool selection, and fallback mechanisms
• Evaluating output consistency across multiple runs
• Testing edge cases such as ambiguous prompts, conflicting goals, and incomplete data
• Validating explainability and reasoning traces where available
• Assessing bias, unsafe outputs, and guardrail effectiveness

Desired Experience

• Exposure to automation (Python, Robot Framework, API automation)
• Experience with AI evaluation frameworks or red-teaming
• Experience in regulated environments (GxP, CSA, 21 CFR Part 11)