QA Agentic AI

Sycamore Informatics Inc.; India; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

Sycamore Informatics Inc. seeks an Agentic AI QA Analyst to validate quality, reliability, safety, and correctness of Agentic AI systems, including testing autonomous agents, multi-step reasoning flows, tool orchestration, and human-in-the-loop controls.

The role requires strong analytical skills to evaluate non-deterministic AI behavior, with responsibilities spanning planning, memory handling, tool calling, and risk-based testing in regulated environments where applicable.

Qualifications

• 2–5 years total experience in QA for AI/Agentic systems.
• Bachelor’s degree in CS, EE, Math or related discipline is preferred.
• Experience with AI evaluation frameworks or red-teaming is a plus.

Responsibilities

• Test agentic AI workflows involving planning, reasoning, action, and reflection loops
• Validate autonomous decision-making paths and goal completion behavior
• Test prompt chaining, context persistence, and memory handling
• Validate tool calling (APIs, databases, search, RPA, internal services)
• Design tests for hallucinations, partial failures, retries, and recovery logic
• Validate human-in-the-loop checkpoints, approvals, overrides, and escalation paths
• Perform functional, exploratory, regression, and scenario-based testing
• Document defects with reproducible prompts, agent state, tools invoked, and outputs
• Support risk-based testing and compliance requirements (CSA / GxP where applicable)
• Under the Sycamore Product and coordinate with different teams to get the work done

Skills

Agentic AI testing

Multi-agent testing

Prompt testing

Non-deterministic outputs evaluation

LLM tool calling

API testing

Log/trace inspection

RAG understanding

Education

Bachelor’s degree in CS/EE/Math

Tools

REST/GraphQL

Python

Robot Framework

Vector databases

Logging/Tracing tools

Job description

Experience (in years)

2 - 5 years (total experience - expected for the role)

Education

e.g., Bachelor’s degree, preferably in Computer Science, Electrical Engineering, Physics, Math or any other related discipline.

Summary

The Agentic AI QA Analyst is responsible for validating the quality, reliability, safety, and correctness of Agentic AI systems. This includes testing autonomous agents, multi-step reasoning flows, tool orchestration, and human-in-the-loop controls. The role requires strong analytical skills to evaluate non-deterministic AI behavior.

Responsibilities

• Test agentic AI workflows involving planning, reasoning, action, and reflection loops
• Validate autonomous decision-making paths and goal completion behavior
• Test prompt chaining, context persistence, and memory handling
• Validate tool calling (APIs, databases, search, RPA, internal services)
• Design tests for hallucinations, partial failures, retries, and recovery logic
• Validate human-in-the-loop checkpoints, approvals, overrides, and escalation paths
• Perform functional, exploratory, regression, and scenario-based testing
• Document defects with reproducible prompts, agent state, tools invoked, and outputs
• Support risk-based testing and compliance requirements (CSA / GxP where applicable)
• Under the Sycamore Product and coordinate with different teams to get the work done

Technical Skills – Agentic AI Testing

• Understanding of Agentic AI architectures (Planner–Executor, ReAct, AutoGPT-style agents)
• Experience testing multi-agent systems and agent-to-agent communication
• Prompt testing and prompt-chain validation techniques
• Testing non-deterministic outputs using evaluation heuristics and acceptance bands
• Experience validating LLM tool calling and function calling
• API testing (REST/GraphQL) for agent-integrated services
• Ability to inspect logs, traces, and agent execution graphs
• Familiarity with vector databases, embeddings, and retrieval-augmented generation (RAG)
• Understanding of model limitations, hallucination patterns, and AI failure modes

Deep-Dive Testing Experience (Agentic AI)

• Testing planning accuracy and goal decomposition across multi-step tasks
• Validating agent memory (short-term vs long-term) and context carryover
• Testing tool misuse, incorrect tool selection, and fallback mechanisms
• Evaluating output consistency across multiple runs
• Testing edge cases such as ambiguous prompts, conflicting goals, and incomplete data
• Validating explainability and reasoning traces where available
• Assessing bias, unsafe outputs, and guardrail effectiveness

Desired Experience

• Exposure to automation (Python, Robot Framework, API automation)
• Experience with AI evaluation frameworks or red‑teaming
• Experience in regulated environments (GxP, CSA, 21 CFR Part 11)