Engineer

550 Eli Lilly Services India Pvt Ltd; Bengaluru; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

Lilly in Bengaluru is seeking a Senior Data Engineer - Data Experience Engineering to design end-to-end data pipelines in the Databricks + AWS Lakehouse. You will translate scientific and regulatory users’ needs into reusable data products, apply AI-assisted tooling, and uphold DataOps standards across Bronze/Silver/Gold layers.

You will mentor engineers, establish patterns, and collaborate with clinical data managers, biostatisticians, and solution architects to accelerate data availability for

Qualifications

• Must have hands-on data engineering delivering production-grade pipelines in enterprise environments.
• Strong Python and SQL skills with PySpark/Spark for distributed processing.
• Experience with cloud data platforms (Databricks on AWS) and data modelling across lakehouse patterns.

Responsibilities

• Lead end-to-end data pipelines across Bronze/Silver/Gold Lakehouse layers on Databricks + AWS.
• Apply DataOps practices: automated testing, IaC, CI/CD, observability; drive architecture reviews.
• Build AI-assisted, self healing pipelines and enable automated data quality checks and lineage capture.
• Mentor engineers, set engineering standards, and translate user needs into data product requirements.

Skills

Python

SQL

PySpark/Spark

Databricks

AWS

Data modelling

Pipeline orchestration

CI/CD

Data observability

CDISC/SDTM/ADaM/OMOP

Education

Bachelor's degree in Computer Science/Data Engineering or related

Tools

Databricks (AWS)

Apache Airflow

Git / CI/CD tools

Great Expectations

Monte Carlo / Acceldata

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work- but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

ABOUT LILLY

At Lilly, everything we do starts with patients. We unite caring with discovery to make life better for people around the world. Headquartered in Indianapolis, Indiana, our global team of over 50,000 employees work with urgency and purpose to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. We bring our best to this work because people depend on it. If you're driven by purpose and determined to make a meaningful difference for patients, we invite you to bring your skill and your commitment to Lilly.

ABOUT TECHNOLOGY@LILLY

At Lilly, technology is not a support function. It is how a global medicine company operates, innovates, and delivers. Lilly in Bengaluru builds the capabilities that make this possible, cloud platforms, AI systems, and automation at enterprise scale, all in service of a purpose that makes this technology work genuinely distinctive, from advancing drug discovery to enabling connected clinical trials to keeping a global medicine company running at the standard patients deserve.

ABOUT THE BUSINESS FUNCTION

At Lilly, CTI-MD Data is the data heavy organization building data products and intelligence layer for Medicine Development - owning 100+ marketplace data products, hundreds of clinical pipelines, and the governance foundation behind Lilly's regulatory submissions, data locks, and patient safety reporting. We're modernizing to a unified, AI enabled context-ready Lakehouse, and building a team where engineers own domain outcomes end to end.

ROLE

The Senior Data Engineer - Data Experience Engineering is an experience-led senior individual contributor who starts from evidence about how scientists, statisticians, clinical data managers, safety, and regulatory teams actually discover, trust, and consume data - and then builds the pipelines and data products that answer those needs. The role works upstream of delivery, replacing assumption with structured user evidence, and carries that evidence through to production across Bronze / Silver / Gold Lakehouse pipelines on Databricks and AWS. It converts business and user needs into technology direction - capability requirements, evaluation criteria, and how enterprise platforms are selected and configured - and holds the engineering depth to prove those recommendations in working code. AI-assisted solutions are the expected way of working across both halves of the role, and as a senior engineer this role sets standards, mentors others, and packages proven patterns as reusable assets across Clinical and Non-Clinical squads.

KEY RESPONSIBILITIES

• Experience-Led Discovery & Data Product Definition Plan and run mixed-methods research - contextual inquiry, interviews, usability testing, surveys, and behavioral analytics - with scientific, clinical, safety, regulatory, and business communities to determine which data products should exist and why. Map user journeys and data workflows across Clinical and Non-Clinical domains, surfacing friction, workarounds, and unmet needs, and analyse product telemetry alongside qualitative findings so behaviour and stated need are read together. Translate evidence into prioritised, decision-ready requirements, data contracts, and design principles, distinguishing genuine capability gaps from usability, adoption, and change-management gaps.
• Data Platform & Pipeline Engineering Independently design, build, and own end-to-end pipelines spanning Bronze / Silver / Gold Lakehouse layers on the Databricks + AWS ecosystem, owning reliability, performance, and cost for assigned data products. Lead the design & build of metadata-driven, reusable pipeline frameworks that reduce time-to-data, and set and enforce squad-level engineering standards and patterns. Apply DataOps practices - automated testing, CI/CD, infrastructure-as-code (Terraform / Bicep), observability - and lead architecture and design reviews, surfacing risks and trade-offs early.
• AI & Automation-Driven Engineering Build self-healing, AI-augmented pipelines using anomaly detection and automated remediation to reduce manual intervention and improve reliability. Apply LLM- and agent-based tooling to accelerate pipeline development, testing, and documentation, and to automate data quality checks, schema drift detection, and lineage capture. Use AI-assisted methods as the default for research and synthesis - study design, transcript analysis, thematic synthesis, opportunity sizing - applying rigorous human judgment to guard against over-generalised conclusions.
• DaaS Delivery, Technology Direction & Platform Evaluation Deliver assigned domain data products from requirement definition through SLA-backed production operation, implementing data contracts and publishing documentation in the enterprise catalog for genuine self-service access. Define and track experience measures (task success, time to insight, trust in data, adoption) and establish baselines that make improvement visible over time. Shape how enterprise platforms and third-party solutions are evaluated, selected, and configured - user-centred criteria, proofs of concept, fit-gap analysis, and evidence-based build / buy / configure recommendations supported by working prototypes. Advance Lilly's federated data mesh model by applying domain data-ownership and governance patterns within assigned products.
• Technical Leadership & Stakeholder Alignment Mentor engineers through code review, pairing, and practical guidance, and establish the quality bar for how user evidence is gathered, interpreted, and reflected in build decisions. Package proven engineering patterns, research methods, and accelerators as reusable components for adoption across Clinical and Non-Clinical squads. Build effective relationships with clinical data managers, biostatisticians, product owners, and solution architects, communicating technical trade-offs, findings, and delivery status to technical and non-technical audiences alike. Maintain alignment between business strategy and the technology landscape over time, supporting standards-compliant pipelines that accelerate availability of clinical trial data for regulatory submission.

QUALIFICATIONS REQUIRED

• Required - (Must-Have) Bachelor's degree in Computer Science, Data Engineering, or a related discipline. Substantial hands-on data engineering, independently delivering production-grade pipelines and data platforms at senior level. Direct exposure to Clinical and/or Non-Clinical data users and their workflows.
• Required - Engineering Depth (Must-Have) Strong Python and SQL; solid working proficiency in PySpark / Spark for distributed data processing. Cloud data platforms (Databricks - AWS); data modelling across dimensional, normalised, and medallion / lakehouse patterns. Pipeline orchestration (Databricks Workflows, Apache Airflow) and CI/CD & DevSecOps practices: Git, Azure DevOps / GitHub Actions, automated testing, IaC. Data observability and quality frameworks (Great Expectations, Monte Carlo, Acceldata, or equivalent). Healthcare / clinical data standards awareness: CDISC, SDTM, ADaM, or OMOP.
• Required - Experience Evidence & Technology Direction (Must-Have) Proven, hands-on command of mixed-methods research applied to real enterprise or scientific user populations, including journey maps, personas, and opportunity frameworks for expert user groups. Demonstrated experience converting business and user needs into technology direction — capability requirements, decision criteria, and roadmap input that engineering teams have adopted. Hands-on experience evaluating, selecting, and configuring enterprise platforms or third-party solutions, including fit-gap analysis and build / buy / configure recommendations.
• Required - AI-Assisted Ways of Working (Must-Have) Current practice of using AI-assisted tools (LLM-based assistants, Copilot-style tools, agentic workflows) as the default method for both engineering and synthesis work — not occasional use. Experience building self-healing, self-monitoring pipelines that automate error detection, retries, and remediation with minimal manual intervention. Experience building automation and reusable frameworks that other engineers, researchers, or product teams can adopt directly. Preferred Data mesh and data product architecture principles; API-first data design; data marketplace, catalog, and self-service analytics adoption. GxP / 21 CFR Part 11 compliance in a validated data environment. MLOps / feature store integration supporting AI/ML model development. Real-world evidence (RWE) or patient-generated data pipeline experience.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly is an EEO/Affirmative Action Employer and does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly At Lilly we strive to ensure our employees are part of a team that cares about them and our shared purpose of making life better for those around the world. How do we do this? We continue to look for ways to include, innovate, accelerate and deliver while maintaining integrity, excellence and respect for people. We hope that you seek to join us on our journey as we create medicine and deliver improved outcomes for patients across the globe! #WeAreLilly