TestOps Engineer - Agentic AI

XenonStack Pvt. Ltd.; Mohali; On-site; INR 1,200,000 - 1,800,000; Full time

Benefits offered by this job

Office in Mohali

Job summary

XenonStack Mohali seeks a TestOps Engineer – Agentic AI with strong Python automation, LLM evaluation, and agentic workflow testing. You will build evaluation harnesses, golden datasets, and regression frameworks for AI agents across ElixirData, ElixirClaw, and NexaStack.

This engineering-focused role emphasizes automated testing and CI/CD integration; you will enable teams to test their components, implement release quality gates, and contribute to unsafe-output prevention and AI safety testing.

Qualifications

• 2–8 years of QA Automation, SDET, MLOps or related engineering roles.
• Hands-on Python automation and test framework development.
• Experience with LLM evaluation tools and agentic AI workflows.
• CI/CD integration and release engineering.

Responsibilities

• Build and maintain evaluation harnesses for LLM-powered agents and agentic workflows.
• Integrate offline evaluations for prompts, models, tools, and retrieval into CI/CD release gates.
• Measure agent quality, reliability, latency, and execution cost against defined thresholds.
• Create and version golden datasets, synthetic conversations, and edge-case scenario libraries for customer workflows.
• Validate agent trajectories, tool selection, execution sequences, and tool arguments.

Skills

Python automation

pytest

API testing

CI/CD

LLM evaluation

Agentic AI

Regression testing

Git

Adversarial testing

AI safety testing

Tools

Docker

Kubernetes

LangChain

LangSmith

Promptfoo

Ragas

DeepEval

Job description

TESTOPS ENGINEER – AGENTIC AI
Location: XenonStack office, Sector 75, STPI, Mohali (Chandigarh Tricity), work from office
Experience: 2–8 years | Employment: Full time, XenonStack Private Limited | Team: AgentOps / Agentic AI Engineering
Role Level: X2 – Workflow

ABOUT XENONSTACK
XenonStack is a Data and AI Foundry for Agentic Systems, enabling enterprises to design, deploy, operate, and scale intelligent agents across digital and physical environments. We build enterprise-grade platforms across the agentic stack:

• Akira AI – Reasoning and agent orchestration.
  
• ElixirData – Agentic analytics intelligence. Explainable, decision-centric analytics for measurable business outcomes.
  
• NexaStack – Agentic infrastructure automation. Secure, compliant AI deployment across cloud, edge, and on-prem.
  
• MetaSecure – Trust, compliance and defense. Continuous assurance with AI-BOMs, risk scoring, and agentic security.
  

Our mission is to accelerate the world's transition to AI + Human Intelligence by making agentic systems reliable, responsible, and enterprise-ready.

ABOUT THE ROLE
We are looking for a TestOps Engineer – Agentic AI with strong expertise in Python automation, LLM evaluation, agentic workflow testing, and CI/CD integration.

You will design and maintain shared evaluation infrastructure, golden datasets, automated quality gates, and regression frameworks for AI agents across ElixirData, ElixirClaw, and NexaStack.

This is an engineering-focused TestOps role, not a traditional manual QA position. You will build the testing systems and enable engineering teams to test their own components.

ROLE MISSION
Make every AI agent release measurable, repeatable, reliable, and safe to ship through automated evaluations, statistical quality measurements, and continuous regression testing.

KEY RESPONSIBILITIES

• Build and maintain reusable evaluation harnesses for LLM-powered agents and agentic workflows.
  
• Integrate offline evaluations for prompts, models, tools, and retrieval into CI/CD release gates.
  
• Measure agent quality, reliability, latency, and execution cost against defined thresholds.
  
• Create and version golden datasets, synthetic conversations, and edge-case scenario libraries for customer workflows.
  
• Validate agent trajectories, tool selection, execution sequences, and tool arguments.
  
• Develop contract tests for MCP servers, APIs, and external tools used by agents.
  
• Handle non-deterministic AI behavior using repeated evaluations, pass@k scoring, and statistical thresholds.
  
• Implement LLM-as-a-Judge evaluations and calibrate results against human-labelled datasets.
  
• Conduct adversarial testing for prompt injection, jailbreaks, data leakage, and policy violations.
  
• Execute regression evaluations during model, prompt, provider, and dependency upgrades.
  
• Establish release quality gates and validate releases against agreed quality and safety thresholds.
  
• Convert production incidents into automated regression tests and maintain shared quality dashboards.
  
• Provide reusable test templates and evaluation playbooks to engineering teams.
  

HOW WE WORK WITH AI
AI is part of the daily workflow at XenonStack, not an add-on.

• At our Mohali/Chandigarh office, AI tools are open for use: coding assistance, test generation, evaluation design, debugging, documentation, and research.
  
• When working on a client's data or environment, follow the client's compliance and information security policy. Use only approved tools and never expose client data, code, credentials, or configurations to unapproved AI services.
  
• Every AI-generated test, evaluation result, and recommendation must be reviewed, validated, and understood before use.
  
• Come ready to discuss your AI testing stack, how you evaluate non-deterministic outputs, and how you prevent unsafe agent behavior.
  

WHAT SUCCESS LOOKS LIKE

• 100% of agent releases go through automated evaluation gates.
  
• Every active customer workflow has a versioned golden dataset with agreed scenario coverage.
  
• Every production incident is converted into a regression test within 5 days.
  
• Zero critical AI safety findings reach production.
  
• All engineering pods adopt the shared evaluation harness rather than maintaining separate testing scripts.
  
• Quality dashboards provide measurable visibility into agent reliability, latency, cost, and evaluation results.
  

MUST-HAVE SKILLS

• Python and test automation: Strong Python programming, pytest, reusable test framework development, API testing, automated regression testing, and Git.
  
• LLM evaluation frameworks: Hands-on experience with DeepEval, Promptfoo, Ragas, LangSmith, or equivalent evaluation tools.
  
• Agentic AI: Understanding of RAG, LLMs, tool calling, agent execution trajectories, and agent frameworks.
  
• Agent integration testing: Experience validating APIs, MCP servers, tool contracts, and multi-step agent workflows.
  
• Evaluation techniques: Golden datasets, synthetic test generation, pass@k, statistical thresholds, repeated runs, and LLM-as-a-Judge evaluation.
  
• CI/CD and release engineering: Integrating automated evaluations into CI/CD pipelines and enforcing release quality gates.
  
• AI safety testing: Knowledge of prompt injection, jailbreaks, data leakage, adversarial scenarios, and policy validation.
  
• Engineering fundamentals: Debugging, root-cause analysis, test coverage, reproducibility, and maintainable automation code.
  

GOOD-TO-HAVE SKILLS

• LangChain, LangGraph, or other agent orchestration frameworks.
  
• Claude, Azure OpenAI, Amazon Bedrock, or other LLM providers.
  
• Multi-agent system testing and agent-to-agent interaction validation.
  
• Docker, Kubernetes, and cloud-native deployment environments.
  
• AI observability, tracing, quality dashboards, and production monitoring.
  
• AI security, red-teaming, responsible AI, and governance frameworks.
  
• Experience with model and prompt versioning, benchmarking, and cost optimisation.
  

CANDIDATE PROFILE

• 2–8 years of experience in QA Automation, SDET, MLOps, or related engineering roles, including hands-on LLM evaluation.
  
• Proven experience developing Python-based automation frameworks and integrating them into CI/CD workflows.
  
• Practical understanding of LLM applications, RAG systems, tool calling, and agentic AI architectures.
  
• Ability to evaluate non-deterministic systems using statistical metrics rather than relying only on binary pass/fail checks.
  
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline; demonstrated hands-on experience is equally important.
  
• Strong analytical thinking, debugging, communication, and cross-team collaboration skills.
  

We value engineers who build scalable testing systems rather than simply execute test cases. Be ready to explain how you evaluated an AI agent, designed golden datasets, handled inconsistent outputs, and prevented regressions from reaching production.

Traditional manual QA experience alone is not sufficient for this role.

TERMS

• Location: Work from our Mohali (Chandigarh Tricity) office.
  
• Full time; immediate joiners or short notice periods preferred.
  
• Background verification (BGV) is mandatory.
  
• Compensation as per company policy and budget band.
  

HOW WE WORK
We value deep work, clarity, and execution over activity. You will be expected to clarify, plan, execute, and escalate with options, not wait to be told. Ownership sits with the individual, and outcomes are measured, not effort.

OUR VALUES
Agency – Be self-directed and proactive | Taste – Sweat the details and build with precision | Ownership – Take responsibility for outcomes | Mastery – Commit to continuous learning and growth | Impatience – Move fast and challenge inertia | Customer Obsession – Always put the customer first

WHY JOIN US

• Build evaluation and testing infrastructure for enterprise-grade Agentic AI platforms.
  
• Work on LLM evaluation, agent reliability, AI safety, and production-grade quality engineering.
  
• Collaborate with AgentOps, engineering, security, and observability teams.
  
• AI-first environment: work alongside the agentic platforms we build.
  
• Great Place to Work-Certified, 14+ years, headquartered in Mohali.