Senior Manager - Testing & AI Quality Engineering

Ramco Systems; Chennai; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Ramco Systems in Chennai, India seeks an Associate Director / Senior Manager for Testing & AI Quality Engineering to lead AI quality across copilots, LLM workflows, and agentic software. This role drives test strategy, governance, and scalable QA transformation in a global software enterprise.

Responsibilities include AI safety testing, evaluation harnesses, and building accelerators using Claude Code and related tooling.

Qualifications

• Bachelor’s or Master’s degree in Computer Science or Engineering.
• 18+ years in test engineering, including 10+ years as a QA manager on complex products.
• 5+ years leading 50+ engineers in a global setting.
• Hands-on tools built with Claude Code or equivalent.
• Production experience testing LLM and agentic software.

Responsibilities

• Own test strategy for copilots, assistants, LLM workflows, RAG pipelines, and multi-step agents across Ramco products.
• Test AI quality with golden datasets, eval harnesses, LLM-as-judge and pass thresholds.
• Run regression for models and prompts with version pinning and drift detection.
• Lead AI red teaming and AI safety testing including prompts and permissions.
• Define AI-in-QA strategy and build testing accelerators with agentic tools.

Skills

Claude Code

Agent engineering

LLM testing

QA management

AI safety testing

Automation testing

Education

Bachelors or Masters in CS/Engineering

Tools

LangSmith

Langfuse

Braintrust

DeepEval

Ragas

promptfoo

Job description

Associate Director / Senior Manager – Testing & AI Quality Engineering

Location: Chennai, India | Experience: 18+ Years | Job Type: Full-time | Function: Quality Assurance & Test Engineering | Reports To: Head – Quality Engineering

RAMCO CONGLOMERATE

Ramco Systems is part of the $1 billion+ diversified Ramco Group, a legacy enterprise with deep roots in industries such as cement, textiles, engineering, software, energy, and biotech. With over 75 years of heritage and innovation, the Ramco Group is a global powerhouse that blends traditional business values with modern technologies.

Ramco Systems is the Group’s technology arm, delivering next-gen enterprise software across Global Payroll, HR, Aviation MRO, ERP, and Logistics. Headquartered in Chennai, India, Ramco serves over 1,000 customers across 35+ countries and is recognized for its AI/ML-based, bot-enabled, and cloud-native solutions.

Our platform is designed for digital operations—trusted by governments, Fortune 500 companies, and high-growth enterprises alike.

If you’re looking to be part of a company that combines innovation, global exposure, and a mission to disrupt legacy systems with intelligent enterprise applications—Ramco is the place to be.

KEY RESPONSIBILITIES

Testing AI & Agentic Software (core focus)

• Own test strategy for copilots, assistants, LLM workflows, RAG pipelines, and multi-step agents across Ramco products.
• Build evaluation-driven quality: golden datasets, eval harnesses, LLM-as-judge with human calibration, and pass thresholds agreed with Product before release.
• Run regression for models and prompts — version pinning, drift detection, re-baselining on model upgrades, and A/B release comparison.
• Test RAG quality (retrieval precision and recall, groundedness, citation correctness, hallucination rate) and agents (tool selection, trajectories, memory, retries, loops, handoffs, MCP and tool contracts).
• Validate blast radius of autonomous action — permission scoping, approval gates, human-in-the-loop, idempotency, and rollback.
• Lead AI red teaming (prompt injection, jailbreaks, data exfiltration, PII and tenant leakage) and AI non-functional quality (latency, token cost, concurrency, fallback).
• Extend quality into production through tracing, online evals, and drift monitoring, feeding failures back into regression.
• Ensure responsible-AI readiness — bias, explainability, and audit evidence against EU AI Act, NIST AI RMF, and ISO/IEC 42001.

AI-Led Testing Transformation

• Define the AI-in-QA strategy across test design, automation, test data, triage, RCA, and release readiness — including where AI is deliberately not used.
• Personally build testing accelerators with Claude Code or comparable agentic tools, and establish agent engineering practices (prompts, context files, skills, sub-agents, MCP integrations to ALM and CI, shared workflow repositories).
• Drive AI-assisted test design with mandatory human review, NL-to-script automation (Playwright, Selenium), self-healing locators, risk-based test selection, synthetic and masked test data, and AI defect intelligence.
• Govern AI-generated artefacts — review gates, traceability, IP hygiene, secure data handling, and human ownership of every released test asset.
• Measure AI impact (adoption, cycle time, coverage, escaped defects, cost per release), retire what shows no value, and reskill 40–50 engineers in prompt, context, and agent engineering.

Quality Strategy & Organisational Leadership

• Set the quality vision and multi-year roadmap with Product, Engineering, Data Science, Support, and Delivery.
• Run Quality Maturity Assessments with the Quality CoE and ensure QMS adherence.
• Balance Dev and QA capacity, and re-baseline QA sizing as AI gains land.
• Own the operating budget, including AI tooling and token spend, with ROI tracking.

Test Engineering, Automation & Non-Functional Quality

• Manage strategies across functional automation, API, performance, accessibility, and security testing.
• Drive testability and observability, including the instrumentation AI features need to be evaluable.
• Build simulation environments with mocked model endpoints for deterministic CI.
• Run proofs of concept comparing AI testing platforms against internal builds, and plan end-to-end testing across products and partners.
• Track productivity, defect leakage, automation coverage, AI adoption and, for AI features, eval pass rates and hallucination trends.
• Report licensing, financials, delivery, and quality to executives, and resolve production issues with global stakeholders.
• Lead managers, leads, and engineers across performance, automation, accessibility, security, and AI quality.
• Define career paths for AI quality and test tooling engineers, and build a culture of experimentation and psychological safety.

WHAT SUCCESS LOOKS LIKE

• AI features released against defensible, pre-agreed evidence rather than subjective confidence.
• Non-deterministic behaviour caught in evals before it reaches a customer workflow.
• AI used in QA where it demonstrably earns its place, and retired where it does not.
• Escaped defects trending down across products while release cadence goes up.
• A test organisation whose engineers can build their own tooling with coding agents.
• Responsible-AI evidence available when a customer, auditor, or regulator asks for it.

Performance in this role is measured against the following indicators. Targets are set annually with the Head of Quality Engineering and reviewed on the cadence noted.

REQUIRED EXPERIENCE & SKILLS

• Bachelor’s or Master’s degree in Computer Science or Engineering.
• 18+ years in test engineering, including 10+ years as a QA manager on complex products.
• 5+ years leading 50+ engineers including managers, in a matrixed global MNC, with a proven shift from QA to test engineering to AI-augmented testing.
• Hands-on tools built with Claude Code or equivalent, with concrete examples of users and savings; agent engineering across context files, skills, sub‑agents, and MCP.
• Production experience testing LLM and agentic software — copilots, RAG, multi-step workflows — not lab‑only.
• Eval frameworks for non-deterministic systems: golden datasets, LLM‑as‑judge with human calibration, and defensible release criteria; RAG quality measurement.
• AI safety testing — prompt injection, jailbreaks, tool data leakage, and agent permission scoping.
• Tooling: LangSmith, Langfuse, Braintrust, DeepEval, Ragas, promptfoo, and agent‑run tracing; judgement on AI economics and when deterministic tests are the better answer.

Core Test Engineering

• Functional and non-functional testing, Agile and Scrum, SQL validation, and quality standards and metrics for strategic programmes.
• Automation and performance tooling: Playwright, Selenium, Cucumber, Postman, JMeter, LoadRunner; REST, SOAP, GraphQL, and event‑driven APIs.
• ALM: Jira/Xray, Azure DevOps, TestRail, Zephyr, with APIs for agent‑driven integration.
• ISTQB Advanced Test Manager certification.

PREFERRED QUALIFICATIONS

• AI quality gates embedded in CI/CD.
• Exposure to EU AI Act, NIST AI RMF, and ISO/IEC 42001.
• Human-in-the-loop models for agents operating on enterprise data.
• AI enablement contributions — training, guilds, hackathons.
• AI certification such as CT-AI; PMP or PRINCE2 an advantage.

WHY JOIN RAMCO SYSTEMS

• Lead quality for AI assistants, copilots, and autonomous agents embedded in enterprise workflows used across 35+ countries.
• A genuine dual mandate — testing AI, and rebuilding testing itself around AI.
• Budget and authority to build tooling rather than only buy it.
• Be part of a fast-evolving, innovation-led enterprise software company.