Principal SRE Engineer

Eli Lilly and Company; Hyderabad; On-site; INR 3,800,000 - 7,000,000; Full time

Job summary

Lilly Capability Centre India in Hyderabad seeks a Senior Reliability Engineer focused on self-healing automation for a multi-application production estate. You will author runbooks, codify root-cause patterns, and drive durable fixes with measurable impact.

You’ll partner with the agentic automation team and Operations to validate outcomes, expand coverage, and ensure change-control and audit readiness in a regulated environment.

Qualifications

• 10+ years engineering experience with ownership of self-healing automation or runbook remediation.
• Experience in regulated/audited environments (GxP, SOX, HIPAA, PCI) is highly desirable.
• Hands-on runbook authoring and validation with safe execution and rollback steps.
• Fluent across observability stacks and cloud platforms.

Responsibilities

• Develop self-healing automation and resilience patterns for multi-application estates.
• Design, validate, and maintain remediation runbooks with safe execution order and rollback.
• Lead or contribute to root-cause analysis and blameless postmortems.
• Collaborate with agentic automation teams and Operations on outcome validation.

Skills

SRE practices

Self-healing automation

Runbook authoring

Incident response

Root-cause analysis

Observability

Cloud platforms (AWS/Azure/GCP)

Education

Bachelor's degree in Computer Science or related field

Tools

Splunk

Datadog

New Relic

Grafana/Prometheus

Terraform

Kubernetes

AWS

Azure

GCP

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

About The Technology Organization

Technology at Lilly builds and operates mission-critical digital products and platforms that support the discovery, development, and delivery of medicines that make life better for people around the world. Our teams operate in highly regulated, high-availability environments, where operational excellence, reliability, and quality are non-negotiable.

Within Tech@Lilly, the Digital Core organization applies a product, platform, and reliability-first mindset, ensuring that operational capabilities scale sustainably across the enterprise.

About The Team

Tech@Lilly builds and maintains capabilities using pioneering technologies like the most prominent tech companies. What differentiates Lilly IT is that we redefine what's possible through tech to advance our purpose, creating medicines that make life better for people around the world, including data-driven drug discovery, connected clinical trials, resilient enterprise platforms, and intelligent digital operations. We hire the best technology professionals from a variety of backgrounds, so they can bring an assortment of knowledge, skills, and diverse thinking to deliver creative solutions in every area of our business.

The Digital Core team leads Lilly's transformation into the Digital and AI era. They inspire digitally empowered teams to new ways of working and accelerate innovation and agility. This team powers and advances the entire company by building and maintaining world-class technology capabilities and platforms.

Lilly Capability Centre India (LCCI), Hyderabad, is Lilly's premier Global Technology Hub, harnessing data, AI, analytics, and digital solutions to revolutionize healthcare and improve patient outcomes worldwide.

The Reliability Engineering team is the engineering-first function that owns the stability, observability, and operational quality of a multi-application production estate. It operates in close partnership with the engineering team that builds the agentic automation platform, and is in active transition from human-executed operations to engineering-led, agent-assisted reliability.

Role summary

You build the self-healing automation, author the runbooks, and turn root-cause analysis into durable engineering fixes that let the production estate heal itself instead of paging a human. You work within the standards the Senior Principal SRE Engineer sets — SLOs, error budgets, observability — and you're the one who encodes them into working automation and documented procedure.

This is a hands-on individual-contributor role focused on execution and codification rather than cross-estate reliability strategy. You decide, in partnership with the Senior Principal SRE Engineer, which recurring patterns warrant a self-healing investment versus a documented manual runbook, and you build whichever is right.

You are an individual contributor. You do not manage people. You partner daily with the Senior Principal SRE Engineer, the agentic automation engineering team, and Operations on validating outcomes. Success is measured by self-healing coverage, runbooks authored and adopted, reduced recurrence of known failure modes, and the safety record of every automation you sign off.

What you'll be doing

• Self-healing automation & resilience patterns
• Design and build self-healing automation — circuit breakers, graceful degradation, automated remediation — for the failure modes that recur most across the estate.
• Run resilience or chaos testing to validate that self-healing patterns behave correctly before they're trusted in production.
• Continuously expand self-healing coverage as new failure modes are identified and proven safe to automate.
• Partner with the Senior Principal SRE Engineer on which failure modes justify self-healing investment versus a documented manual runbook.
• Runbook authorship & validation
• Author and validate the remediation runbooks for the production estate: safe execution order, rollback steps, and exception handling for every documented fix.
• Keep the runbook library current as systems, dependencies, and failure modes evolve, retiring runbooks that no longer apply.
• Define and apply the graduation criteria that let a runbook move from human-executed to agent-assisted to autonomous.
• RCA to durable fix
• Lead or contribute to root-cause analysis for significant incidents, and drive the blameless postmortem process to a durable engineering fix — not just a narrative.
• Convert recurring incident patterns into codified runbooks and, where appropriate, self-healing automation.
• Track fix effectiveness against recurrence, and elevate to the Senior Principal SRE Engineer when a fix needs broader engineering investment.
• Participate in high-severity incident response, including acting as incident commander for escalations within your area.
• Partnership with agentic automation & operations
• Partner with the Agentic Automation Engineering team on which fixes are safe to hand off as agent-assisted remediations, and on the confidence thresholds and human-in-the-loop boundaries that keep them safe.
• Sign off on agent graduation criteria (accuracy over volume, zero P1/P2 caused) before an automation moves to a higher autonomy tier.
• Partner with Operations on outcome validation, feeding what's learned back into the runbook library and self-healing patterns.
• Incident response & regulated-environment practice
• Ensure runbooks and self-healing automation meet Lilly's change-control, audit, and validated-environment standards.
• Document procedures so that audit evidence falls out of normal operation, not a special exercise.
• Mentor other reliability and automation engineers on runbook quality and self-healing design.
• Contribute proven patterns back to the broader reliability practice, in partnership with the Senior Principal SRE Engineer and Senior Architect.

How you will succeed

• Be recognized as the engineer who turns incidents into durable fixes, not repeat pages.
• Demonstrate measurable growth in self-healing coverage and runbook adoption, with falling recurrence of known failure modes.
• Maintain a clean safety record: automations you sign off don't cause P1/P2 incidents.
• Build runbooks and automation that make good practice the default, not a personal habit.

What you should bring

Required

• 10+ years of progressive engineering experience, with meaningful time as a Site Reliability Engineer, Production Engineer, or equivalent, including hands-on ownership of self-healing automation or runbook-driven remediation for a multi-application production estate.
• Production reliability experience in a regulated or audited environment (GxP, SOX, HIPAA, PCI, or equivalent), including familiarity with change-control discipline, audit evidence, and validated-system constraints.
• Hands-on experience authoring and validating runbooks: safe execution order, rollback steps, and exception handling for real remediation procedures.
• Deep, hands-on fluency across the SRE technical stack: observability (Splunk, Datadog, New Relic, or Grafana/Prometheus); infrastructure-as-code (Terraform); CI/CD pipeline hardening; Kubernetes and container platforms; and at least one major public cloud (AWS, Azure, or GCP).
• Practical experience designing self-healing patterns (circuit breakers, graceful degradation, automated remediation) and validating them before they're trusted in production.
• Track record of contributing meaningfully to root-cause analysis and blameless postmortems, converting findings into durable engineering fixes measured by reduced recurrence rather than narrative quality.
• Bachelor's degree or higher in Computer Science, Information Technology, or a closely related engineering field.

Preferred

• Hands-on experience designing self-healing automation and running chaos engineering or resilience-testing programs (AWS Fault Injection Service, Gremlin, LitmusChaos, or equivalent) tied to measurable reliability gains.
• Deep AWS fluency across reliability-relevant services (EKS, ECS, Lambda, CloudWatch, X-Ray, Systems Manager, Route 53), and familiarity with AWS Well-Architected Reliability Pillar.
• Experience with AIOps or agent-assisted operations, including designing the guardrails, confidence thresholds, and human-in-the-loop boundaries that make automated remediation safe in production.
• Experience defining or signing off on autonomy/graduation criteria for automation moving toward self-sufficiency.
• Prior experience building or scaling a runbook library across multiple applications or teams.
• Experience operating in pharma, healthcare, financial services, or other regulated industries.

Leadership expectations

• Treats every RCA as unfinished until it produces a durable fix, not just a narrative.
• Combines engineering rigor with operational pragmatism in deciding what gets automated and what stays manual.
• Leads through what they build and how they write, not through org-chart authority.
• Comfortable telling a team that a fix isn't ready for autonomy yet, and showing them what's missing.
• Treats runbook quality and knowledge-sharing as a first-class outcome, not an afterthought.

Additional information

Availability to work flexible work hours is/may be required. This team supports continuous operations and may require non-standard work hours, including some work on weekends and holidays. Appropriate adjustments in benefits will be provided for employees working non-standard hours where applicable.

This is an onsite role based in Hyderabad. Candidates should be open to working different shifts when required to align with global delivery partners.

At Lilly, caring is not only what we do for patients. It is how we work. We believe the people who dedicate themselves to making medicines better deserve an environment that makes their lives better too, one where they are supported, respected, and given the space to do their best work. This is not just a policy. It is who we are.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly