Test Architect / Tech Lead - CTI MD Tech (GRA/GSC)

Eli Lilly and Company; Bengaluru; On-site; INR 2,600,000 - 5,200,000; Full time

Job summary

Eli Lilly and Company in Bengaluru invites an experienced Test Architect / Tech Lead to shape the automation framework and lead end-to-end delivery for Global Regulatory Affairs and Scientific Communications. You will design architecture, build reusable components, and drive AI evaluation for generative systems, partnering with engineering, product, and compliance to ensure quality and scale.

This role requires seniority, strong leadership, and a track record delivering high-impact tech

Qualifications

• Proven track record architecting and scaling test automation frameworks across multiple concurrent programs.
• Experience designing application architecture and leading end-to-end delivery of key technical initiatives as a tech lead.
• Experience driving technical direction for AI-driven or agentic platforms, not solely test execution.
• Experience setting quality/validation standards enforced consistently across distributed or vendor-delivered teams.
• Track record of operating with seniority across functional boundaries in ambiguous, fast-scaling transformation environments.

Responsibilities

• Reusable Test Framework Development — design, build, and own reusable test automation frameworks and shared components.
• Application Architecture & Design — define technical approach and integration patterns for GRA/GSC platforms.
• Delivery Ownership — act as tech lead for end-to-end architecture, build, test, and release.
• Automation Enablement — drive test automation to replace manual and duplicated testing.
• AI & Agentic Quality Engineering — build evaluation frameworks for generative AI and agentic systems.
• Test Initiative Leadership — scope, prioritize, and deliver automation improvements across teams.

Skills

Test automation

Quality engineering

AI evaluation

Agentic systems

Stakeholder management

Observability

Tools

Python

JavaScript/TypeScript

RAGAS

DeepEval

LLM-as-a-Judge

Veeva RIM

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.Test Architect / Tech Lead - CTI MD Tech (GRA/GSC)Global Regulatory Affairs (GRA)

• Global Scientific Communications (GSC)
• CTI-MD Tech@Lilly

The Test Architect / Tech Lead (R4), GRA/GSC is accountable for building the test automation frameworks and reusable testing components that drive quality engineering across GRA and GSC, with particular depth in evaluating and validating generative AI and agentic systems. The role also carries tech lead accountability: designing and building the application architecture for GRA/GSC platforms and owning delivery of key initiatives end to end, not solely test strategy. This is a single R4 path role by design: test standards, automation architecture, and application architecture need to be set once and held consistently rather than fragmented across roles without the seniority to drive strategic evolution end to end. The role owns the build-out of reusable frameworks and components, drives test automation initiatives, leads delivery of key technical initiatives, and manages stakeholders to ensure quality and AI evaluation rigor scale consistently as GRA/GSC's agentic platforms mature.Path/Level: R4 (Test Architect / Tech Lead)Key Responsibilities

• Reusable Test Framework Development — design, build, and own reusable test automation frameworks and shared components (libraries, harnesses, fixtures, evaluation pipelines) that can be adopted consistently across GRA and GSC rather than rebuilt per program.
• Application Architecture & Design — design and build the application architecture for GRA/GSC platforms as a tech lead — defining technical approach, component design, and integration patterns alongside the test and quality strategy.
• Delivery Ownership — act as tech lead for key technical initiatives, owning end-to-end delivery — from architecture and build through testing and release — and holding accountability for outcomes, not just quality gates.
• Automation Enablement — drive test automation efforts across GRA and GSC, identifying opportunities to replace manual and duplicated testing effort with shared, scalable automation.
• AI & Agentic Quality Engineering — build and own evaluation frameworks for generative AI and agentic systems — groundedness validation, hallucination detection, relevance/accuracy testing, and RAG and LLM evaluation — using tools such as RAGAS, DeepEval, and LLM-as-a-Judge approaches.
• Test Initiative Leadership — define and drive test initiatives end to end — scoping, prioritizing, and delivering automation and quality-engineering improvements across concurrent teams and workstreams.
• Stakeholder Management — partner with engineering leads, product owners, and quality/compliance stakeholders to align on architecture, delivery plans, test strategy, and adoption of shared frameworks.
• Observability & Trace Analysis — establish agent observability and trace-analysis practices to diagnose failures in multi-agent and agentic workflows, and feed findings back into architecture, framework, and test design.
• Synthetic Test Data & Coverage Strategy — build synthetic test data generation capabilities and apply risk-based prioritization to determine test depth and coverage across critical workflows.
• Talent Development & Mentorship — mentor test engineers and technical leads, building a team capable of operating independently on architecture, framework design, automation, and AI evaluation.

Technical & Strategic Expertise AreasApplication & Solution Architecture

• AI Evaluation Frameworks (RAGAS, DeepEval, LLM-as-a-Judge)
• Groundedness Validation
• Hallucination Detection
• Relevance & Accuracy Testing
• Generative AI Testing
• Agentic AI Testing
• RAG Validation
• LLM Evaluation
• Prompt Engineering & Context Validation
• Multi-Agent Systems
• Agentic Frameworks
• AI Quality Engineering
• End-to-End Test Automation
• API & Integration Testing
• Agent Observability & Trace Analysis
• Synthetic Test Data Generation
• Python
• JavaScript/TypeScript

Basic Qualifications

• Proven track record architecting and scaling test automation frameworks across multiple concurrent programs.
• Demonstrated experience designing application architecture and leading end-to-end delivery of key technical initiatives as a tech lead.
• Demonstrated experience driving technical direction for AI-driven or agentic platforms, not solely test execution.
• Experience setting quality/validation standards enforced consistently across distributed or vendor-delivered teams.
• Track record of operating with seniority across functional boundaries in ambiguous, fast-scaling transformation environments.

Additional Qualifications / Competencies

• Experience in both Global Regulatory Affairs and Scientific Communications, or comparable regulated pharma functions.
• Familiarity with Regulatory Content and Submission Management, Veeva RIM, and regulatory platforms.
• Experience with document and content management ecosystems.
• Background in pharma and GxP-regulated environments.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response. Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status. #WeAreLilly