Senior CE Data Engineer

100 Eli Lilly and Company; Indianapolis (IN); On-site; USD 63,000 - 150,000; Full time

Benefits offered by this job

401(k) Plan

Pension

Vacation benefits

Medical, dental, vision benefits

Job summary

Eli Lilly and Company is seeking a CE Data Engineer to design and operationalize a Databricks-based data platform within the CE trust boundary. You will implement medallion pipelines, data contracts, identity crosswalks, and isolation controls, collaborating closely with the CE Data Architect and Analytics Engineers to deliver governed data products.

The role emphasizes production-grade pipelines, data quality, auditing, and scalable solutions with strong emphasis on PHI handling and security.

Qualifications

• Bachelor's degree in Computer Science, Engineering, IT, or similar.
• Advanced SQL and Python; Databricks / Unity Catalog fluency.
• Authorized to work in the United States on a full-time basis.

Responsibilities

• Design and build scalable Databricks pipelines across bronze→silver→gold.
• Apply Databricks capabilities like Unity Catalog, Delta Lake, Workflows.
• Implement PHI classification and identity crosswalk logic.
• Build ODCS data contracts and enforce quality, SLA, lineage.
• Ensure PHI isolation with row/column security and masking.
• Own end-to-end pipeline lifecycle from conception to production.
• Develop CI/CD pipelines for data, contracts, policies, and artifacts.
• Automate data ingestion and onboarding of new sources.
• Collaborate with Architects, Product Owner, and engineers.
• Support governance, security, and compliant data consumption paths.

Skills

Advanced SQL

Python

Databricks Unity Catalog

Data governance

RBAC/ABAC security

CI/CD pipelines

MDM / identity resolution

HIPAA knowledge

GitHub Actions

Education

Bachelor's degree in CS/Engineering or related field

Tools

Delta Lake

Reltio

Auth0

Datavant

Terraform

ODCS

Job description

Organization Overview

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—and it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

Position Summary

The CE Data Engineer builds and operationalizes the CE Pharmacy data platform on Databricks — implementing the medallion pipelines, data contracts, identity crosswalk logic, and isolation controls that the CE Data Architect designs. This is a hands‑on engineering role: writing production pipelines, evaluating and applying Databricks capabilities and integration patterns, and holding the line on data quality, testing, and monitoring so the platform is reliable, auditable, and built to scale. Works in close, day‑to‑day partnership with the CE Data Architect and the Associate Director of CE Data Engineering — turning architectural intent into running code — and with Analytics Engineers and the DataHub Product Owner to ship governed data products.

Key Responsibilities

• Design and build scalable, efficient Databricks pipelines implementing the CE Architect's canonical data models across the medallion (bronze → silver → gold), scoped entirely within the CE trust boundary.
• Evaluate and apply Databricks capabilities and integration patterns — Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, ingestion connectors — selecting the right tool for each pipeline given performance, cost, and scalability constraints.
• Implement the identity crosswalk and PHI classification logic defined by the CE Architect (Reltio MDM, Auth0/Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, Transcend), safeguarding merge/split integrity in code.
• Build to ODCS data contracts — implement schema, quality, freshness/SLA, and lineage requirements as enforced pipeline logic, not documentation.
• Implement row/column-level security, masking, and tokenization boundaries so PHI isolation is enforced at the platform layer, in partnership with the Policy-as-Code Engineer's OPA/Rego policies.
• Own end-to-end pipeline development lifecycle — requirements to prototyping to production deployment and maintenance — for CE data products, in partnership with Analytics Engineers and the DataHub Product Owner.
• Build and maintain CI/CD pipelines (Github Actions [CI], Git-based promotion dev → test → prod) for CE data, contract, policy, and agent artifacts.
• Automate data ingestion and product creation to reduce manual pipeline maintenance and onboarding time for new CE data sources.
• Partner with the CE Data Architect on reference architecture and patterns, providing implementation feedback that keeps designs buildable and performant at scale.
• Build the data pipelines that lets CE Skills and agents consume governed data — ensuring PHI classification and consent travel with the data into agentic consumption paths.

Basic Requirements

Bachelor's degree in Computer Science, Engineering, Information Technology, or similar degree. Experience in data engineering, with a focus on building production data pipelines and data products. Advanced SQL and Python; hands‑on Databricks / Unity Catalog fluency (notebooks, Delta Lake, catalog/schema/grants, Workflows). Qualified applicants must be authorized to work in the United States on a full‑time basis. Lilly will not provide support for or sponsor work authorization or visas for this role, including but not limited to F-1 CPT, F-1 OPT, F-1 STEM OPT, J-1, H-1B, TN, O-1, E-3, H-1B1, or L-1.

Additional Skills / Preferences

• Experience defining and executing data ingestion pipelines at enterprise scale.
• Working knowledge of data governance, classification, and access control (RBAC/ABAC, row- and column-level security).
• Proficiency with Git-based CI/CD workflows (Git Actions or equivalent) for versioned data and pipeline artifacts.
• Excellent problem‑solving skills; ability to translate architectural designs into working, tested pipelines.
• Good communication and collaboration skills — able to work effectively with architects, product owners, and multi‑functional partners.
• Experience in regulated or healthcare data environments; familiarity with HIPAA, PHI handling, and covered-entity constructs.
• Data contract frameworks (ODCS or comparable) and policy-as-code (OPA/Rego) awareness.
• Tokenization / de‑identification pattern implementation (Datavant or comparable).
• PySpark and transformation-as-code (DLT); test frameworks (pytest, DQX / Great Expectations).
• Infrastructure-as-code (Terraform) for provisioning Unity Catalog objects and grants.
• MDM / identity resolution implementation experience (Reltio or comparable).
• Experience with Agile/Scrum methodologies and Jira.
• Experience building or integrating AI Skills and agents against governed data platforms.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.

Employee Resource Groups

Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include:

• Africa
• Middle East
• Central Asia (AMECA)
• Black Employees at Lilly (BE@Lilly)
• Chinese Culture Network (CCN)
• EnAble
• Evolve
• Lilly Indian Network (LIN)
• Organization of Latinx at Lilly (OLA)
• Pride (LGBTQ+ Allies)
• Veterans Leadership Network (VLN)
• Women’s Initiative for Leading at Lilly (WILL)

Compensation and Benefits

Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is $63,000 - $149,600 Full‑time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance).

In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well‑being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).

• eligibility to participate in a company-sponsored 401(k)
• pension
• vacation benefits
• eligibility for medical, dental, vision and prescription drug benefits
• flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts)
• life insurance and death benefits
• certain time off and leave of absence benefits
• well‑being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities)

Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.

#WeAreLilly At Lilly we strive to ensure our employees are part of a team that cares about them and our shared purpose of making life better for those around the world. How do we do this? We continue to look for ways to include, innovate, accelerate and deliver while maintaining integrity, excellence and respect for people. We hope that you seek to join us on our journey as we create medicine and deliver improved outcomes for patients across the globe! #WeAreLilly