Lead QA - Agentic AI

Sycamore Informatics Inc.; India; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Sycamore Informatics Inc. is seeking a Senior Agentic AI QA Lead Principle to own the quality strategy for Agentic AI platforms.

This role focuses on defining AI test strategies, evaluating systemic risks, validating autonomous behaviors, and ensuring production readiness of agentic systems. The ideal candidate will design end-to-end test strategies for agentic and autonomous AI systems, lead testing for hallucinations, bias and safety, review validation evidence, and collaborate with

Qualifications

• Deep understanding of agent frameworks and orchestration layers.
• Expertise in testing autonomous workflows and decision policies.
• Experience evaluating planning failures, looping behaviors, and dead-ends.
• Advanced prompt evaluation, adversarial testing, and red-teaming.
• Strong experience testing API-driven and distributed agent architectures.
• Ability to design evaluation metrics for accuracy, reliability, and safety.
• Experience reviewing logs, traces, and telemetry for agent behavior analysis.
• Knowledge of AI governance, model risk management, and guardrails.
• Exposure to automation (Python, Robot Framework, API automation)

Responsibilities

• Define end-to-end testing strategy for agentic and autonomous AI systems
• Design complex test scenarios for reasoning chains, tool orchestration, and multi-agent collaboration
• Lead testing for hallucinations, bias, ethical risks, and unsafe behaviors
• Validate human-in-the-loop controls, auditability, and traceability
• Review and approve AI test cases, defect reports, and validation evidence
• Partner with engineering, product, and compliance teams on risk mitigation
• Mentor QA teams on agentic AI testing techniques
• Provide release recommendations based on risk-based quality assessments
• Define and conduct any scale/performance testing for AI Agents built as well
• Train Sycamore resources with Agentic AI skills where applicable

Skills

Agentic AI testing

Autonomous workflows

Decision policies

Prompt evaluation

Adversarial testing

Logging & telemetry

AI governance

Model risk mgmt

Automation (Python)

Education

Bachelor’s degree

Job description

Experience (in years) - 9 – 15 Years Experience

Education - e.g., Bachelor’s degree, preferably in Computer Science, Electrical Engineering, Physics, Math or any other related discipline. (Education requirements to be entered here)

Summary

The Senior Agentic AI QA Lead Principle owns the quality strategy for Agentic AI platforms. This role focuses on defining AI test strategies, evaluating systemic risks, validating autonomous behaviors, and ensuring production readiness of agentic systems.

• Define end-to-end testing strategy for agentic and autonomous AI systems
• Design complex test scenarios for reasoning chains, tool orchestration, and multi-agent collaboration
• Lead testing for hallucinations, bias, ethical risks, and unsafe behaviors
• Validate human-in-the-loop controls, auditability, and traceability
• Review and approve AI test cases, defect reports, and validation evidence
• Partner with engineering, product, and compliance teams on risk mitigation
• Mentor QA teams on agentic AI testing techniques
• Provide release recommendations based on risk-based quality assessments
• Define and conduct any scale/performance testing for AI Agents built as well
• Train Sycamore resources with Agentic AI skills where applicable

Technical Skills – Advanced Agentic AI Testing

• Deep understanding of agent frameworks and orchestration layers
• Expertise in testing autonomous workflows and decision policies
• Experience evaluating planning failures, looping behaviors, and dead-ends
• Advanced prompt evaluation, adversarial testing, and red-teaming
• Strong experience testing API-driven and distributed agent architectures
• Ability to design evaluation metrics for accuracy, reliability, and safety
• Experience reviewing logs, traces, and telemetry for agent behavior analysis
• Knowledge of AI governance, model risk management, and guardrails
• Exposure to automation (Python, Robot Framework, API automation

Deep-Dive Testing Experience (Senior)

• Designing AI test strategies under uncertainty and non-determinism
• Validating autonomous escalation and self-correction mechanisms
• Testing long-running agents and stateful workflows
• Defining acceptance criteria for probabilistic outputs
• Assessing business and compliance risks of agent decisions
• Establishing testing standards and review processes for AI systems

Desired Experience

• Experience defining AI validation approaches under CSA or GxP
• Automation framework ownership or strategy experience
• Experience in AI governance, audits, or regulatory inspections
• Working with Any LLMs like Claude with aws bedrock