Senior Automation Architect-Gen AI

Cognizant; Hyderabad, Chennai, Bengaluru; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

Cognizant seeks a Senior Architect to design and lead our testing and evaluation ecosystem, bridging UI/API automation with GenAI validation. You will own LLM evaluation architecture using Ragas and DeepEval while modernizing enterprise Java Selenium test suites.

Strong Python for AI/ML workflows is essential. The role emphasizes architecture, mentorship, and collaboration with Data Scientists, ML Engineers, and Product Managers across multiple India locations.

Qualifications

• 12+ years of total experience in software test architecture, automation engineering, or SDET roles.
• 3+ years of dedicated experience working with GenAI, LLM validation, or MLOps frameworks.
• Proven ability to author architectural blueprints, technical roadmaps, and whitepapers for enterprise stakeholders.
• Experience with vector databases (ChromaDB, Pinecone, Milvus) and LLM orchestration tools like LangChain or LlamaIndex.
• Familiarity with alternative modern testing tools like Playwright.

Responsibilities

• Design and implement robust evaluation pipelines for LLM-powered applications (RAG pipelines, agents, and chatbots) using Python, Ragas, and DeepEval.
• Define core GenAI metrics including faithfulness, answer relevance, context recall, context precision, and hallucination detection.
• Architect automated, continuous evaluation systems integrated into CI/CD pipelines to monitor LLM performance regression.
• Establish guardrails and synthetic data generation strategies to stress-test LLM application robustness and safety.
• Architect and scale enterprise-grade web automation frameworks using Java and Selenium WebDriver.
• Design hybrid automation frameworks (BDD/Cucumber, TestNG/JUnit, Page Object Model) with parallel execution and cross-browser testing.
• Lead migration/integration between legacy Java-based testing and modern Python-based AI evaluation tools.
• Optimize locator strategies, handle dynamic web elements, and minimize flaky tests across large regression suites.
• Provide technical leadership, mentorship, and architectural governance to QA, AI, and DevOps teams.
• Integrate evaluation/testing frameworks with CI/CD platforms (Jenkins, GitLab CI, GitHub Actions) and cloud environments (AWS, Azure, GCP).
• Collaborate with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria for AI products.

Skills

Python for AI/ML

Java Selenium

Playwright TS

GenAI frameworks

CI/CD

Ragas/DeepEval

LangChain/LlamaIndex

Docker/Kubernetes

REST/GraphQL

Tools

Selenium WebDriver

Playwright

Docker

Kubernetes

LangChain

LlamaIndex

ChromaDB

Pinecone

Milvus

GitHub Actions

Job description

Role Overview

We are seeking a visionary and hands‑on Senior Architect to design, scale, and lead our next‑generation testing and evaluation ecosystem. In this role, you will bridge the gap between traditional UI/API automation and cutting‑edge GenAI/LLM validation.

You will own the architecture for evaluating Large Language Model (LLM) applications using Ragas and DeepEval, while maintaining and modernizing our enterprise Java Selenium test automation suites. The ideal candidate has deep expertise in Python for AI/ML workflows and a proven track record of designing robust enterprise automation frameworks from scratch.

Key Responsibilities

1. LLM Evaluation & GenAI Architecture

• Design and implement robust evaluation pipelines for LLM-powered applications (RAG pipelines, agents, and chatbots) using Python, Ragas, and DeepEval.
• Define core GenAI metrics including faithfulness, answer relevance, context recall, context precision, and hallucination detection.
• Architect automated, continuous evaluation systems integrated directly into CI/CD pipelines to monitor LLM performance regression.
• Establish guardrails and synthetic data generation strategies to stress‑test LLM application robustness and safety.

2. Enterprise UI/API Automation & Modernization

• Architect and scale enterprise‑grade web automation frameworks using Java and Selenium WebDriver.
• Design hybrid automation frameworks (BDD/Cucumber, TestNG/JUnit, Page Object Model) that support parallel execution and cross‑browser testing.
• Lead the migration or integration efforts between legacy Java‑based testing infrastructure and modern Python‑based AI evaluation tools.
• Optimize locator strategies, handle dynamic web elements, and minimize flaky tests across large‑scale regression suites.

3. Strategy, Leadership & DevOps

• Provide technical leadership, mentorship, and architectural governance to QA, AI, and DevOps engineering teams.
• Integrate evaluation and testing frameworks with CI/CD platforms (Jenkins, GitLab CI, GitHub Actions) and cloud environments (AWS, Azure, or GCP).
• Collaborate closely with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria for AI products.

Required Skills & Qualifications

Technical Core

• Programming: Advanced, production‑level proficiency in both Python (for AI/ML workflows), Playwright(Typescript) and Java (for enterprise automation).
• LLM Evaluation: Hands‑on experience architecting frameworks with Ragas, DeepEval, or TruLens. Deep understanding of prompt engineering and RAG trilemma.
• UI Automation: Expert‑level knowledge of Java Selenium WebDriver, Grid, and advanced automation patterns.
• APIs & Data: Strong experience testing RESTful/GraphQL APIs and handling data serialization (JSON, XML).
• CI/CD & DevOps: Experience with Docker, Kubernetes, and setting up automated testing pipelines.

Experience & Leadership

• 12+ years of total experience in software test architecture, automation engineering, or SDET roles.
• 3+ years of dedicated experience working with GenAI, LLM validation, or MLOps frameworks.
• Proven ability to author architectural blueprints, technical roadmaps, and whitepapers for enterprise stakeholders.
• Experience with vector databases (ChromaDB, Pinecone, Milvus) and LLM orchestration tools like LangChain or LlamaIndex.
• Familiarity with alternative modern testing tools like Playwright.

Location- Chennai/Coimbatore/Hyderabad/Bangalore/Pune

Notice Period- Max 30 days (Officially)

Mode of Interview-Virtual

Date of Interview- 17/10/2026

Note: Please do not apply for this role if your skillset is not relevant to this role or your notice period is above 45 days since this role is critical.