Engineer

550 Eli Lilly Services India Pvt Ltd; Bengaluru; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Lilly seeks a Data Engineer in Bengaluru to design, build, and operate production-grade data pipelines across Bronze to Gold lakehouse architectures on Databricks and AWS. The role emphasizes AI/automation, self-healing pipelines, and data contracts for enterprise governance.

You will contribute to a federated data mesh, mentor junior engineers, and collaborate with clinical data teams to enable rapid, compliant data delivery for analytics and regulatory submissions.

Qualifications

• Bachelor's degree in Computer Science, Data Engineering, or a related discipline.
• Hands-on data engineering experience delivering production-grade pipelines and data platforms.
• Strong Python and SQL; working proficiency in PySpark/Spark for distributed data processing.
• Cloud data platforms: Databricks & AWS.
• Data modelling: dimensional, normalised, and lakehouse patterns.
• Data pipeline orchestration (Databricks Workflows, Apache Airflow).
• CI/CD and DevSecOps practices: Git, Azure DevOps / GitHub Actions, automated testing, IaC.
• Hands-on experience applying AI and automation to pipeline engineering.
• Data observability and quality frameworks familiarity.

Responsibilities

• Design, build, and own end-to-end data pipelines spanning Bronze / Silver / Gold Lakehouse layers on Databricks + AWS.
• Own pipeline reliability and performance for assigned data products.
• Contribute to metadata-driven, reusable pipeline frameworks to reduce time-to-data for new use cases.
• Apply DataOps with automated testing, CI/CD, and IaC; incorporate AI-assisted development where applicable.
• Build self-healing, AI-augmented pipelines with anomaly detection and automated remediation.
• Use LLM- and agent-based tooling to accelerate pipeline development, testing, and documentation.
• Automate data quality checks, schema drift detection, and lineage capture with AI-assisted rules.
• Design intelligent orchestration that adapts based on data signals and history.
• Prototype opportunities to apply generative AI to eliminate repetitive tasks.
• Deliver data products from requirement definition through production operation.
• Publish data contracts and documentation for self-service access.

Skills

Python

SQL

PySpark

Databricks

AWS

Data modelling

Data pipelines

CI/CD

AI in data engineering

Data observability

Education

Bachelor's degree in Computer Science or related

Tools

Databricks

AWS

Airflow

Git

Azure DevOps

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

ABOUT LILLY

At Lilly, everything we do starts with patients. We unite caring with discovery to make life better for people around the world. Headquartered in Indianapolis, Indiana, our global team of over 50,000 employees work with urgency and purpose to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. We bring our best to this work because people depend on it. If you're driven by purpose and determined to make a meaningful difference for patients, we invite you to bring your skill and your commitment to Lilly.

ABOUT TECHNOLOGY@LILLY

At Lilly, technology is not a support function. It is how a global medicine company operates, innovates, and delivers. Lilly in Bengaluru builds the capabilities that make this possible, cloud platforms, AI systems, and automation at enterprise scale, all in service of a purpose that makes this technology work genuinely distinctive, from advancing drug discovery to enabling connected clinical trials to keeping a global medicine company running at the standard patients deserve.

ABOUT THE BUSINESS FUNCTION

At Lilly, CTI-MD Data is the data heavy organization building data products and intelligence layer for Medicine Development — owning 100+ marketplace data products, hundreds of clinical pipelines, and the governance foundation behind Lilly's regulatory submissions, data locks, and patient safety reporting. We're modernizing to a unified, AI enabled context-ready Lakehouse, and building a team where engineers own domain outcomes end to end.

ROLE

The Data Engineer is an experienced individual contributor within the Data Engineering team who independently designs, builds, and operates production-grade data pipelines and platforms serving clinical and non-clinical data consumers. Working with a strong degree of autonomy, this role owns end-to-end delivery of moderately complex data products, applies and helps refine squad-level engineering standards, and begins extending its impact by mentoring less-experienced engineers and sharing knowledge across the team. This role brings a strong AI and automation skillset — using generative AI, agentic tooling, and intelligent automation to build self-monitoring, self-healing pipelines that reduce manual effort and accelerate delivery. This role contributes directly to Lilly's federated data mesh vision, translating it into reliable, well-governed data services that support regulatory submissions, scientific analytics, and AI/ML programmes.

KEY RESPONSIBILITIES

• Data Platform & Pipeline Engineering Design, build, and own end-to-end data pipelines spanning Bronze / Silver / Gold Lakehouse layers on the Databricks + AWS ecosystem.
• Own pipeline reliability and performance for assigned data products; apply squad-level engineering standards and patterns consistently.
• Contribute to the design of metadata-driven and reusable pipeline frameworks that reduce time-to-data for new use cases.
• Apply DataOps practices including automated testing, CI/CD, infrastructure-as-code (Terraform / Bicep), and observability tooling, incorporating AI-assisted development where applicable.
• AI & Automation-Driven Pipeline Engineering Build self-healing, AI-augmented pipelines that use anomaly detection and automated remediation to reduce manual intervention and improve reliability.
• Apply LLM- and agent-based tooling (e.g., Copilot-style code generation, agentic workflow orchestration) to accelerate pipeline development, testing, and documentation.
• Automate data quality checks, schema drift detection, and lineage capture using AI-assisted rules and pattern recognition rather than static, manually maintained logic.
• Design intelligent orchestration that dynamically adapts pipeline execution (retries, scaling, routing) based on data volume, quality signals, and historical run patterns.
• Identify and prototype opportunities to apply generative AI and automation to eliminate repetitive engineering tasks across the squad's pipeline portfolio.
• Data as a Service (DaaS) Delivery Deliver assigned domain data products — from requirement definition through SLA-backed production operation.
• Implement data contracts between producer and consumer domains, ensuring semantic consistency and discoverability.
• Publish and maintain data product documentation in the enterprise catalog, enabling self-service access for scientists, statisticians, and business analysts.
• Support Lilly's evolution toward a federated data mesh model by applying domain data-ownership and governance patterns within assigned products.
• Technical Contribution & Mentorship Participate in architecture and design reviews, contributing informed technical recommendations and identifying risks early.
• Mentor and support junior engineers on the squad, providing code review feedback and practical guidance on pipeline design.
• Share learnings from project work and external research through team knowledge-sharing sessions.
• Evaluate new tools and techniques relevant to assigned platform components and propose pragmatic adoption where they add value.
• Stakeholder Collaboration Build effective working relationships with clinical data managers, biostatisticians, and business analysts to clarify and refine data product requirements.
• Communicate technical trade-offs and delivery status clearly to both technical and non-technical stakeholders.
• Contribute to sizing and scoping of platform work, surfacing gaps and risks to the squad's technical lead.
• Support the delivery of automated, standards-compliant pipelines that accelerate the availability of clinical trial data for regulatory submission.

QUALIFICATIONS REQUIRED

• Bachelor's degree in Computer Science, Data Engineering, or a related discipline.
• Years of hands‑on data engineering experience, with a track record of independently delivering production-grade pipelines and data platforms.
• Strong Python and SQL; working proficiency in PySpark / Spark for distributed data processing.
• Cloud data platforms: Databricks & AWS.
• Data modelling: dimensional, normalised, and medallion / lakehouse architecture patterns.
• Data pipeline orchestration (Databricks Workflows, Apache Airflow).
• CI/CD and DevSecOps practices: Git, Azure DevOps / GitHub Actions, automated testing, IaC.
• Hands‑on experience applying AI and automation to pipeline engineering — e.g., LLM-based code generation and testing (Copilot or equivalent), agentic workflow automation, and AI-driven anomaly detection for data quality and pipeline health.
• Experience building self‑healing, self‑monitoring pipelines that automate error detection, retries, and remediation with minimal manual intervention.
• Working knowledge of data observability and quality frameworks (Great Expectations, Monte Carlo, Acceldata, or equivalent).
• Healthcare / clinical data standards awareness: CDISC, SDTM, ADaM, or OMOP.
• Demonstrated experience owning components of a data platform within a cross‑functional product or platform team.
• Preferred Exposure to Data mesh and Data product architecture principles; API‑first data design.
• Experience building or integrating AI agents / LLM‑powered tools into data engineering workflows (e.g., automated pipeline documentation, natural‑language‑to‑pipeline scaffolding, AI code review assistants).
• Familiarity with GxP / 21 CFR Part 11 compliance in a validated data environment.
• Exposure to MLOps / feature store integration supporting AI/ML model development.
• Real‑world evidence (RWE) or patient‑generated data pipeline experience.

EEO AND ACCESSIBILITY STATEMENTS

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly is an EEO/Affirmative Action Employer and does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly At Lilly we strive to ensure our employees are part of a team that cares about them and our shared purpose of making life better for those around the world. How do we do this? We continue to look for ways to include, innovate, accelerate and deliver while maintaining integrity, excellence and respect for people. We hope that you seek to join us on our journey as we create medicine and deliver improved outcomes for patients across the globe! #WeAreLilly