Test Architect / Tech Lead - CTI MD Tech (GRA/GSC)

550 Eli Lilly Services India Pvt Ltd; Bengaluru; On-site; INR 3,000,000 - 5,500,000; Full time

Job summary

Lilly is seeking a senior Test Architect / Tech Lead to design and own reusable test automation frameworks for Global Regulatory Affairs (GRA) and Global Scientific Communications (GSC). You will lead end-to-end delivery of key AI and agentic testing initiatives, establishing observability and synthetic data strategies across multi-agent platforms.

Role emphasizes architecture, AI evaluation (RAGAS, DeepEval, LLM-as-a-Judge), and collaboration with engineering, product, and compliance

Qualifications

• Proven ability to architect and scale test automation frameworks across multiple programs.

Responsibilities

• Reusable Test Framework Development — design, build, and own reusable test automation frameworks and shared components across GRA and GSC.

Skills

Test automation

Framework development

AI evaluation

Generative AI testing

Agentic AI testing

Python

JavaScript/TypeScript

API testing

Tools

RAGAS

DeepEval

LLM-as-a-Judge

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

Test Architect / Tech Lead - CTI MD Tech (GRA/GSC) Global Regulatory Affairs (GRA) Global Scientific Communications (GSC) CTI-MD Tech@Lilly Global Regulatory Affairs (GRA) and Global Scientific Communications (GSC) together own how Lilly's medicines move through health authority review, stay compliant once approved, and get communicated externally — spanning regulatory submissions, labelling, clinical trial disclosures, and scientific publications.

Both functions operate under strict compliance and audit-readiness requirements, and are scaling agentic and generative AI capabilities across their platforms, creating a growing need for consistent, reusable test automation and AI quality engineering practices across GRA and GSC.

The Test Architect / Tech Lead (R4), GRA/GSC is accountable for building the test automation frameworks and reusable testing components that drive quality engineering across GRA and GSC, with particular depth in evaluating and validating generative AI and agentic systems.

The role also carries tech lead accountability: designing and building the application architecture for GRA/GSC platforms and owning delivery of key initiatives end to end, not solely test strategy.

This is a single R4 path role by design: test standards, automation architecture, and application architecture need to be set once and held consistently rather than fragmented across roles without the seniority to drive strategic evolution end to end.

The role owns the build-out of reusable frameworks and components, drives test automation initiatives, leads delivery of key technical initiatives, and manages stakeholders to ensure quality and AI evaluation rigor scale consistently as GRA/GSC's agentic platforms mature.

Key Responsibilities

• Reusable Test Framework Development — design, build, and own reusable test automation frameworks and shared components (libraries, harnesses, fixtures, evaluation pipelines) that can be adopted consistently across GRA and GSC rather than rebuilt per program.
• Application Architecture & Design — design and build the application architecture for GRA and GSC platforms as a tech lead — defining technical approach, component design, and integration patterns alongside the test and quality strategy.
• Delivery Ownership — act as tech lead for key technical initiatives, owning end-to-end delivery — from architecture and build through testing and release — and holding accountability for outcomes, not just quality gates.
• Automation Enablement — drive test automation efforts across GRA and GSC, identifying opportunities to replace manual and duplicated testing effort with shared, scalable automation.
• AI & Agentic Quality Engineering — build and own evaluation frameworks for generative AI and agentic systems — groundedness validation, hallucination detection, relevance/accuracy testing, and RAG and LLM evaluation — using tools such as RAGAS, DeepEval, and LLM-as-a-Judge approaches.
• Test Initiative Leadership — define and drive test initiatives end to end — scoping, prioritizing, and delivering automation and quality-engineering improvements across concurrent teams and workstreams.
• Stakeholder Management — partner with engineering leads, product owners, and quality/compliance stakeholders to align on architecture, delivery plans, test strategy, and adoption of shared frameworks.
• Observability & Trace Analysis — establish agent observability and trace-analysis practices to diagnose failures in multi-agent and agentic workflows, and feed findings back into architecture, framework, and test design.
• Synthetic Test Data & Coverage Strategy — build synthetic test data generation capabilities and apply risk-based prioritization to determine test depth and coverage across critical workflows.
• Talent Development & Mentorship — mentor test engineers and technical leads, building a team capable of operating independently on architecture, framework design, automation, and AI evaluation.

Technical & Strategic Expertise Areas

• Application & Solution Architecture
• AI Evaluation Frameworks (RAGAS, DeepEval, LLM-as-a-Judge)
• Groundedness Validation
• Hallucination Detection
• Relevance & Accuracy Testing
• Generative AI Testing
• Agentic AI Testing
• RAG Validation
• LLM Evaluation
• Prompt Engineering & Context Validation
• Multi-Agent Systems
• Agentic Frameworks
• AI Quality Engineering
• End-to-End Test Automation
• API & Integration Testing
• Agent Observability & Trace Analysis
• Synthetic Test Data Generation
• Python
• JavaScript/TypeScript

Basic Qualifications

• Proven track record architecting and scaling test automation frameworks across multiple concurrent programs.
• Demonstrated experience designing application architecture and leading end-to-end delivery of key technical initiatives as a tech lead.
• Demonstrated experience driving technical direction for AI-driven or agentic platforms, not solely test execution.
• Experience setting quality/validation standards enforced consistently across distributed or vendor-delivered teams.
• Track record of operating with seniority across functional boundaries in ambiguous, fast-scaling transformation environments.

Additional Qualifications / Competencies

• Experience in both Global Regulatory Affairs and Scientific Communications, or comparable regulated pharma functions.
• Familiarity with Regulatory Content and Submission Management, Veeva RIM, and regulatory platforms.
• Experience with document and content management ecosystems.
• Background in pharma and GxP-regulated environments.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions.

If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly At Lilly we strive to ensure our employees are part of a team that cares about them and our shared purpose of making life better for those around the world. How do we do this? We continue to look for ways to include, innovate, accelerate and deliver while maintaining integrity, excellence and respect for people. We hope that you seek to join us on our journey as we create medicine and deliver improved outcomes for patients across the globe! #WeAreLilly