LillyDirect Pharmacy Data Engineer

Eli Lilly and Company; United States; On-site; USD 66,000 - 189,000; Full time

Benefits offered by this job

Company bonus

401(k) and pension

Vacation benefits

Medical, dental, vision

Flexible benefits

Well-being benefits

Job summary

Lilly is seeking a Data Engineer for the LillyDirect Pharmacy Covered Entity Master Data team. You will design, build, and maintain scalable data pipelines and data models to support scientific and business initiatives, ensuring data availability, quality, and integrity.

You will work with the Technical Lead, DevOps, architects, and analysts to implement Delta Lake-based lakehouse architectures on AWS, develop ETL/ELT workflows, and govern sensitive data while driving cost efficiency and

Qualifications

• Bachelor's degree in Computer Science, Information Technology, MIS, or equivalent work experience.
• Overall 5+ years of experience in data engineering using SQL, Python, PySpark, and cloud platforms (AWS, Azure, or GCP).
• 2+ years in Agile environments with hands-on GitHub and CI/CD pipelines for code deployment.
• Hands-on experience with Databricks, Delta Lake, and performance optimization.
• 1+ years of experience with orchestration tools like Airflow or Databricks Workflows.

Responsibilities

• Design, build, and maintain scalable data pipelines for batch and near-real-time processing in Python and PySpark on Databricks and AWS.
• Develop and optimize data models, ETL/ELT workflows, and data integration across systems using Delta Lake and lakehouse patterns.
• Support data integrations with Reltio and Transcend, including loads, extracts, and reconciliation.
• Build data quality, lineage, and reconciliation checks for accurate, auditable data.
• Implement data governance, security, and quality standards across data assets including PII/PHI and consent data.
• Collaborate with Tech Lead, product owners, architects, analysts, and business stakeholders to deliver solutions.
• Own pipelines and work streams from design through production, contributing to architectural decisions.
• Promote best practices in coding, code review, automated testing, and CI/CD pipelines.
• Monitor and improve performance, reliability, and cost of data pipelines; support incidents and root-cause analysis.
• Automate manual processes and identify opportunities to optimize data workflows.

Skills

SQL

Python

PySpark

Cloud platforms

Agile

GitHub

CI/CD

Databricks

Education

Bachelor's degree in CS/IT/MIS or equivalent

Tools

Delta Lake

Reltio

Transcend

Airflow

Terraform

Unity Catalog

Databricks Workflows

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work-but it's work worth doing. If you're driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

We are seeking a highly motivated Data Engineer for our LillyDirect Pharmacy Covered Entity (LDPCE) Master Data team. This role involves designing, building, and maintaining robust and scalable data pipelines and infrastructure to support our critical scientific and business initiatives, ultimately contributing to the delivery of life-changing medicines.

As a Data Engineer for Master Data, you will be responsible for designing, developing, and maintaining the data solutions that ensure the availability, quality, and integrity of data for analysis and/or business transactions for our LillyDirect Pharmacy Covered Entity business. You will design and implement efficient data storage, processing, and retrieval solutions, build data pipelines, and optimize data models, working closely with the Technical Lead, DevOps engineer, architects, and business analysts. You will need strong skills in data integration, acquisition, cleansing, harmonization, and transformation, and you will play a key role in turning raw data into reliable, governed datasets that downstream teams can trust.

What you'll be doing:

• Design, build, and maintain scalable and reliable data pipelines for batch and near-real-time processing in Python and PySpark on Databricks and AWS.
• Develop and optimize data models, ETL/ELT workflows, and data integration across multiple systems and platforms, using Delta Lake and lakehouse (medallion) patterns.
• Support data integrations with Reltio and Transcend, including data loads, extracts, and reconciliation.
• Build data quality, lineage, and reconciliation checks so that profile and consent data is accurate, complete, and auditable.
• Implement data governance, security, and quality standards across data assets, including access controls, masking, and retention for sensitive data (PII/PHI) and consent data.
• Collaborate with the Technical Lead, product owners, architects, analysts, and business stakeholders to understand data requirements and deliver solutions.
• Own pipelines and work streams from design through production, and contribute to architectural and design decisions.
• Promote best practices in coding, code review, automated testing, and deployment, and release code through CI/CD pipelines.
• Monitor, troubleshoot, and improve performance, reliability, and cost of data pipelines, and support production incidents and root cause analysis.
• Automate manual processes and identify opportunities to optimize data workflows and reduce costs.

How You Will Succeed:

• Deliver scalable solutions by designing robust data pipelines and architectures that meet performance and reliability standards.
• Build in privacy and consent by treating consent and data protection as a design principle in pipelines from the start, and by raising risks early.
• Collaborate effectively with cross-functional teams to turn business needs into technical outcomes.
• Share expertise, mentoring junior engineers and adopting best practices in data engineering and cloud technologies.
• Continuously improve systems through automation, performance tuning, and proactive issue resolution.
• Communicate with clarity to ensure alignment across technical and non-technical stakeholders.

Basic Qualifications and Experience Requirement:

• Bachelor's degree in Computer Science, Information Technology, Management Information Systems, or equivalent work experience.
• Overall 5+ years of experience in data engineering using core technologies such as SQL, Python, PySpark, and cloud platforms (AWS, Azure, or GCP) services including Lambda, Glue, EMR S3, Redshift, Athena, and IAM roles/policies.
• 2+ years of experience working in Agile environments, with hands-on experience using GitHub and CI/CD pipelines for code deployment.
• Hands-on experience with Databricks, including workspace configuration, notebook development, Delta Lake, and performance optimization.
• 1+ years of experience with orchestration tools like Airflow or Databricks Workflows for workflow automation.

Additional Skills/Preferences:

• Experience building high-performance, scalable data pipelines following Data Lakehouse, Data Warehouse, and Data Mart standards and deep understanding of ETL/ELT workflows.
• Strong expertise in data modelling, managing large datasets, and implementing secure, compliant data governance practices.
• Working knowledge of data governance, security, and data quality practices, including handling sensitive data (PII/PHI) in a regulated environment.
• Ability to collaborate with business stakeholders to translate key business requirements into scalable technical solutions.
• Strong communication and collaboration skills to work with both technical and non-technical stakeholders; a problem-solving mindset and attention to detail.
• Pharmaceutical or healthcare industry experience, including privacy requirements such as HIPAA.
• Hands-on experience with Reltio (or a comparable MDM platform), including match and merge, survivorship, and integration patterns.
• Familiarity with consent and preference management (for example opt-in/opt-out, purpose, and channel); experience with Transcend or a similar consent platform.
• Familiarity with streaming frameworks such as Kafka, Kinesis, or MSK.
• Experience with Unity Catalog, infrastructure-as-code (for example Terraform), and data quality frameworks (for example Great Expectations or dbt tests).
• Partner with and influence vendor resources on solution development to ensure understanding of data and technical direction for solutions as well as delivery.
• AWS Certified Data Engineer - Associate
• Databricks Certified Data Engineer (Associate or Professional)

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.

Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia (AMECA), Black Employees at Lilly (BE@Lilly), Chinese Culture Network (CCN), EnAble, Evolve, Lilly Indian Network (LIN), Organization of Latinx at Lilly (OLA), Pride (LGBTQ+ Allies), Veterans Leadership Network (VLN) and Women's Initiative for Leading at Lilly (WILL).

Actual compensation will depend on a candidate's education, experience, skills, and geographic location. The anticipated wage for this position is
$66,000 - $189,200

Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly's compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.