Senior Automation Architect

Cognizant; India; Presencial; INR 400.000 - 600.000; Jornada completa

Descripción de la vacante

Cognizant seeks a Senior Architect to design, scale, and lead an evaluation and automation ecosystem bridging UI/API automation with GenAI/LLM validation. You will own LLM evaluation using Ragas and DeepEval while maintaining Java Selenium test suites and driving modernization with Python-based AI workflows.

Key responsibilities include building robust evaluation pipelines, integrating CI/CD and cloud platforms, and guiding QA, AI, and DevOps teams.

Formación

• 12+ years of software test architecture, automation engineering, or SDET roles.
• 3+ years dedicated experience with GenAI, LLM validation, or MLOps frameworks.
• Proven ability to author architectural blueprints, roadmaps, and whitepapers for enterprise stakeholders.
• Experience with vector databases (ChromaDB, Pinecone, Milvus) and LLM orchestration tools like LangChain or LlamaIndex.
• Familiarity with modern testing tools like Playwright.

Responsabilidades

• Design and implement robust evaluation pipelines for LLM-powered applications using Python, Ragas, and DeepEval.
• Architect and scale enterprise-grade web automation frameworks using Java and Selenium WebDriver.
• Lead migration/integration between legacy Java testing infra and modern Python-based AI evaluation tools.
• Define guardrails and synthetic data strategies to stress-test LLM robustness and safety.
• Collaborate with data scientists, ML engineers, and product managers to define acceptance criteria for AI products.

Conocimientos

Python

Java

Playwright (Typescript)

Herramientas

Ragas

DeepEval

TruLens

LangChain

LlamaIndex

Selenium WebDriver

Docker

Kubernetes

Playwright

Descripción del empleo

Senior Architect LLM Evaluation & Automation Frameworks

Role Overview

We are seeking a visionary and hands-on Senior Architect to design, scale, and lead our next-generation testing and evaluation ecosystem. In this role, you will bridge the gap between traditional UI/API automation and cutting-edge GenAI/LLM validation.

You will own the architecture for evaluating Large Language Model (LLM) applications using Ragas and DeepEval, while maintaining and modernizing our enterprise Java Selenium test automation suites. The ideal candidate has deep expertise in Python for AI/ML workflows and a proven track record of designing robust enterprise automation frameworks from scratch.

Key Responsibilities

1. LLM Evaluation & GenAI Architecture

• Design and implement robust evaluation pipelines for LLM-powered applications (RAG pipelines, agents, and chatbots) using Python, Ragas, and DeepEval.
• Define core GenAI metrics including faithfulness, answer relevance, context recall, context precision, and hallucination detection.
• Architect automated, continuous evaluation systems integrated directly into CI/CD pipelines to monitor LLM performance regression.
• Establish guardrails and synthetic data generation strategies to stress-test LLM application robustness and safety.

2. Enterprise UI/API Automation & Modernization

• Architect and scale enterprise-grade web automation frameworks using Java and Selenium WebDriver.
• Design hybrid automation frameworks (BDD/Cucumber, TestNG/JUnit, Page Object Model) that support parallel execution and cross-browser testing.
• Lead the migration or integration efforts between legacy Java-based testing infrastructure and modern Python-based AI evaluation tools.
• Optimize locator strategies, handle dynamic web elements, and minimize flaky tests across large-scale regression suites.

3. Strategy, Leadership & DevOps

• Provide technical leadership, mentorship, and architectural governance to QA, AI, and DevOps engineering teams.
• Integrate evaluation and testing frameworks with CI/CD platforms (Jenkins, GitLab CI, GitHub Actions) and cloud environments (AWS, Azure, or GCP).
• Collaborate closely with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria for AI products.

Required Skills & Qualifications

Technical Core

• Programming: Advanced, production-level proficiency in both Python (for AI/ML workflows), Playwright(Typescript) and Java (for enterprise automation).
• LLM Evaluation: Hands-on experience architecting frameworks with Ragas, DeepEval, or TruLens. Deep understanding of prompt engineering and RAG trilemma.
• UI Automation: Expert-level knowledge of Java Selenium WebDriver, Grid, and advanced automation patterns.
• APIs & Data: Strong experience testing RESTful/GraphQL APIs and handling data serialization (JSON, XML).
• CI/CD & DevOps: Experience with Docker, Kubernetes, and setting up automated testing pipelines.

Experience & Leadership

• 12+ years of total experience in software test architecture, automation engineering, or SDET roles.
• 3+ years of dedicated experience working with GenAI, LLM validation, or MLOps frameworks.
• Proven ability to author architectural blueprints, technical roadmaps, and whitepapers for enterprise stakeholders.
• Experience with vector databases (ChromaDB, Pinecone, Milvus) and LLM orchestration tools like LangChain or LlamaIndex.
• Familiarity with alternative modern testing tools like Playwright.

Location- Chennai/Coimbatore/Hyderabad/Bangalore/Pune

Notice Period- Max 45 days (Officially)