QA Agentic AI

Sycamore Informatics Inc.; India; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Sycamore Informatics Inc. is seeking an experienced AI QA Analyst to validate the quality, reliability, safety, and correctness of Agentic AI systems.

You will test autonomous agents, multi-step reasoning flows, tool orchestration, and human-in-the-loop controls to ensure robust performance. Responsibilities include evaluating planning accuracy, memory carryover, tool usage, and fail-safe behavior across prompts and workflows while documenting defects with reproducible prompts and agent state.

Qualifications

• Exposure to automation (Python, Robot Framework, API automation).
• Experience with AI evaluation frameworks or red-teaming.
• Experience in regulated environments (GxP, CSA, 21 CFR Part 11).

Responsibilities

• Test agentic AI workflows involving planning, reasoning, action, and reflection loops.
• Validate autonomous decision-making paths and goal completion behavior.
• Test prompt chaining, context persistence, and memory handling.
• Validate tool calling (APIs, databases, search, RPA, internal services).
• Design tests for hallucinations, partial failures, retries, and recovery logic.
• Validate human-in-the-loop checkpoints, approvals, overrides, and escalation paths.
• Perform functional, exploratory, regression, and scenario-based testing.
• Document defects with reproducible prompts, agent state, tools invoked, and outputs.
• Support risk-based testing and compliance requirements (CSA / GxP where applicable).
• Under the Sycamore Product and coordinate with different teams to get the work done.

Skills

Agentic AI testing

Multi-agent testing

Prompt testing

LLM tool calling

API testing (REST/GraphQL)

Logs and traces inspection

Vector databases / RAG

Model limitations and failure modes

Python automation

Robot Framework

Education

Bachelor’s degree in a related field

Tools

Python

Robot Framework

Job description

Experience (in years)

- 2 - 5 years (total experience - expected for the role)

Education

- e.g., Bachelor’s degree, preferably in Computer Science, Electrical Engineering, Physics, Math or any other related discipline.

Summary

The Agentic AI QA Analyst is responsible for validating the quality, reliability, safety, and correctness of Agentic AI systems. This includes testing autonomous agents, multi-step reasoning flows, tool orchestration, and human-in-the-loop controls. The role requires strong analytical skills to evaluate non-deterministic AI behavior.

• Test agentic AI workflows involving planning, reasoning, action, and reflection loops
• Validate autonomous decision-making paths and goal completion behavior
• Test prompt chaining, context persistence, and memory handling
• Validate tool calling (APIs, databases, search, RPA, internal services)
• Design tests for hallucinations, partial failures, retries, and recovery logic
• Validate human-in-the-loop checkpoints, approvals, overrides, and escalation paths
• Perform functional, exploratory, regression, and scenario-based testing
• Document defects with reproducible prompts, agent state, tools invoked, and outputs
• Support risk-based testing and compliance requirements (CSA / GxP where applicable)
• Under the Sycamore Product and coordinate with different teams to get the work done

Technical Skills – Agentic AI Testing

• Understanding of Agentic AI architectures (Planner–Executor, ReAct, AutoGPT-style agents)
• Experience testing multi-agent systems and agent-to-agent communication
• Prompt testing and prompt-chain validation techniques
• Testing non-deterministic outputs using evaluation heuristics and acceptance bands
• Experience validating LLM tool calling and function calling
• API testing (REST/GraphQL) for agent-integrated services
• Ability to inspect logs, traces, and agent execution graphs
• Familiarity with vector databases, embeddings, and retrieval-augmented generation (RAG)
• Understanding of model limitations, hallucination patterns, and AI failure modes

Deep-Dive Testing Experience (Agentic AI)

• Testing planning accuracy and goal decomposition across multi-step tasks
• Validating agent memory (short-term vs long-term) and context carryover
• Testing tool misuse, incorrect tool selection, and fallback mechanisms
• Evaluating output consistency across multiple runs
• Testing edge cases such as ambiguous prompts, conflicting goals, and incomplete data
• Validating explainability and reasoning traces where available
• Assessing bias, unsafe outputs, and guardrail effectiveness

Desired Experience

• Exposure to automation (Python, Robot Framework, API automation)
• Experience with AI evaluation frameworks or red-teaming
• Experience in regulated environments (GxP, CSA, 21 CFR Part 11)