Associate Director – RWD Data Engineering

Eli Lilly and Company

Indiana

On-site

USD 137,000 - 200,000

Full time

47 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

401(k) plan
Pension
Vacation benefits

Job summary

Eli Lilly and Company, headquartered in Indianapolis, IN, seeks an Associate Director for RWD Data Engineering to lead cloud-native data pipelines and AI-enabled tools on the Databricks Lakehouse. You will enable HEOR, SDIA, and clinical teams with validated data products built from claims and EHR data, ensuring quality and reproducibility at scale.

The role reports to the Senior Director – Real World Data Engineering & Architecture within HEOR Central and requires collaboration with multiple

Qualifications

  • Hands-on data engineering in healthcare/RWD
  • Experience with Databricks and Delta Lake
  • Proficiency in Python, SQL, and Spark
  • Familiarity with AI coding assistants and frontier AI models
  • Genuine collaboration with cross-functional teams

Responsibilities

  • Design, develop, and deploy cloud-native data pipelines on Databricks.
  • Write, test, and document Python/SQL/R code with AI assistance.
  • Deliver data products through full development lifecycle in a Lakehouse.
  • Build ETL/ELT pipelines for large-scale claims and EHR data.
  • Maintain data assets within lakehouse using Delta Lake & Unity Catalog.
  • Create reusable transformation logic for healthcare datasets.
  • Optimize SQL and distributed workloads for performance and cost.
  • Translate requirements into reproducible data extraction logic.
  • Implement data quality and validation frameworks across products.
  • Apply CI/CD standards and version control in daily work.
  • Develop interactive analytic apps (Dash, Shiny, Streamlit).
  • Explore frontier AI models and Databricks Model Serving for data workflows.

Skills

Python
SQL optimization
AI coding assistants
Spark

Education

Bachelor’s degree in CS/Math/Engineering

Tools

Databricks
Delta Lake
Git
CI/CD

Job description

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life‑changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us.

Purpose

Job Description

The Associate Director, RWD Data Engineering is a hands‑on technical role that builds and delivers the engineering vision for Lilly’s real‑world data (RWD) infrastructure set by the Senior Director – RWD Engineering & Architecture. This individual designs, codes, tests, and deploys the scalable, cloud‑native pipelines, data products, and analytic applications that allow HEOR, SDIA, Statisticians, Medical, and Clinical teams to generate evidence faster, more reproducibly, and at greater scientific depth than is possible through traditional vendor engagements.

This role requires deep, practical knowledge of secondary healthcare data, with a particular focus on administrative claims and electronic health records (EHR), and of the Databricks Lakehouse platform used to engineer it. The Associate Director turns raw RWD into validated, analysis‑ready data products; automates data ingestion, quality checks, and product creation; and builds AI‑enabled tools and analytic applications that put those data products directly in the hands of scientists. This individual applies AI coding assistants and frontier AI models as an everyday part of software development, and is accountable for the correctness, performance, and reproducibility of the code they deliver. The ideal candidate is an expert builder who pairs strong engineering fundamentals with rigor in data quality and a passion for leveraging data to drive business outcomes.

This position reports to the Senior Director – Real World Data Engineering & Architecture within HEOR Central and works in close partnership with CIA, HEOR, SDIA, Statisticians, Medical, and Clinical teams.

Responsibilities
  • undefined
  • Design, develop, and deploy cloud‑native data products and high‑throughput data pipelines on Databricks that transform raw real‑world data into scalable, reliable, analysis‑ready assets supporting analytics, reporting, and evidence generation.
  • Leverage AI coding assistants and agentic development tools (e.g., Claude Code, GitHub Copilot, Cursor, Databricks Assistant) to write, refactor, test, and document Python, SQL, and R code, applying rigorous human review, testing, and validation to every AI‑generated contribution.
  • Deliver assigned RWD data products through the full development lifecycle, from technical requirements and prototyping through production deployment and optimization, ensuring scalability, reliability, performance, and reproducibility within the established Databricks Lakehouse environment.
  • Build, optimize, and maintain ETL/ELT ingestion and transformation pipelines for large‑scale secondary RWD, with a particular focus on administrative claims and complex EHR data, handling data volumes ranging from tens of millions to billions of records.
  • Build and maintain data assets within the established lakehouse architecture (e.g., medallion bronze/silver/gold patterns) using Databricks, Delta Lake, Unity Catalog, and Databricks Workflows to produce versioned, partitioned, and audit‑ready data assets.
  • Write and maintain reusable, version‑controlled transformation logic incorporating healthcare coding and terminology standards (e.g., ICD‑10/ICD‑9, NDC, RxNorm, SNOMED, CPT‑4, HCPCS, LOINC) to produce domain‑level datasets such as demographics, diagnoses, treatments, procedures, encounters, and labs.
  • Optimize SQL and distributed processing workloads (e.g., Databricks SQL and Spark jobs) for performance and compute cost across very large datasets, reading query plans and applying partitioning, liquid clustering/Z‑ordering, predicate pushdown, caching, and other optimization strategies appropriate to analytical workloads.
  • Translate analytic and research requirements into reproducible data extraction and transformation logic, supporting cohort construction, temporal logic, and consistent reuse of RWD across teams.
  • Design, implement, and run data quality and validation frameworks for every data product, including automated checks for completeness, conformance, plausibility, and referential integrity; reconciliation of record counts and distributions against source; regression testing between data refreshes; and monitoring and alerting that surface issues before they reach analytical teams.
  • Apply the team’s version control and CI/CD standards in daily work, including Git branching, pull requests, peer code review, automated unit and data tests, and deployment via Databricks Asset Bundles, ensuring all code is version‑controlled, tested, and deployable in a reproducible manner.
  • Build and deploy interactive analytic applications (e.g., Dash, R Shiny, Streamlit, Databricks Apps) that allow HEOR scientists, statisticians, and medical teams to explore data products, review cohort counts and data quality results, and self‑serve common analyses.
  • Build AI‑enabled solutions that operate directly on governed data products using frontier AI models (e.g., Anthropic Claude, OpenAI GPT, Google Gemini) and Databricks Model Serving, such as natural‑language query of data products, AI agents that execute data workflows, and retrieval over data dictionaries, code lists, and documentation; evaluate these solutions for accuracy and ensure they meet Lilly’s privacy, security, and responsible‑AI requirements.
  • Collaborate closely with data scientists, statisticians, and other technical partners to understand technical requirements, and contribute to documentation of RWD engineering standards, transformation templates, code list repositories, and pipeline performance guidelines.
  • Serve as the technical point of contact for assigned data sources and data products, advising collaborators on appropriate use, structural limitations, join strategies, and performance considerations; contribute to source‑specific training materials for HEOR scientists, SDIA, and statisticians.
  • Contribute to an inclusive culture where producing and maintaining high‑quality data is a core discipline.
Technical Skills
Core Engineering Skills
  • Expert SQL optimization on Databricks SQL and Spark SQL (with Redshift, Snowflake, or BigQuery a plus): complex joins, window functions, query plan analysis, query tuning, and compute cost management on billion‑row tables.
  • Python fluency: pandas, PySpark, Polars for large‑scale data manipulation; workflow orchestration with Apache Airflow, Prefect, or Dagster for production pipeline scheduling and monitoring — including Databricks Workflows for orchestrating multi‑task jobs within the Lakehouse.
  • Distributed computing: Apache Spark (PySpark), Dask, or Ray — ability to write, tune, and debug distributed jobs processing multi‑terabyte datasets across partitioned cloud storage, including Databricks clusters with auto‑scaling and spot instance optimization.
  • Databricks expertise: hands‑on, production development on Databricks (AWS, Azure, or GCP), including notebooks, Workflows, Delta Live Tables, Unity Catalog, SQL warehouses, and cluster configuration; experience with AWS (S3, Glue) or Azure (ADLS, ADF) object storage and services.
  • Delta Lake / Apache Iceberg: time travel, schema evolution, upsert/merge operations, partition optimization — building versioned, ACID‑compliant data assets at scale; Delta Lake experience ideally hands‑on within the Databricks Lakehouse platform using Delta Live Tables (DLT) for declarative pipeline authoring.
  • ETL/ELT tooling: AWS Glue, dbt, dbt‑databricks, or equivalent for building tested, documented transformation pipelines with automated data quality checks; familiarity with Databricks Asset Bundles (DABs) for packaging and deploying notebooks, jobs, and DLT pipelines as code.
  • Version control and DevOps: daily use of Git and GitHub/Azure DevOps (branching strategies, pull requests, code review, merge conflict resolution), CI/CD pipelines (GitHub Actions, Azure DevOps), and Docker, maintaining all code as version‑controlled, testable, and deployable artifacts; experience deploying to Databricks via Asset Bundles, CLI, or REST API.
  • Data modeling: building star/snowflake schemas, OMOP‑compliant structures, and flat domain long files optimized for analytical workloads in a research context, materialized as managed Delta tables within Databricks Unity Catalog with appropriate access controls and lineage tracking.
Analytic Application Development
  • Building and deploying interactive analytic applications with Dash (Plotly), R Shiny, Streamlit, or Databricks Apps, connected to Databricks SQL warehouses and Unity Catalog with appropriate authentication and access controls.
  • Front‑end fundamentals sufficient to deliver usable, performant applications for scientific users, including caching, pagination, and efficient query patterns against large tables.
  • R programming: advanced R development for data engineering and analytics, including building and deploying production R Shiny dashboards; writing and maintaining internal R packages that standardize how teams connect to, query, and analyze data products (e.g., shared connection functions to Databricks, cohort‑building utilities, and reusable analytic functions); and applying good package development practices such as roxygen2 documentation, testthat unit tests, version control, and renv for reproducible environments.
Data Quality & Validation
  • Designing automated data quality frameworks using tools such as Delta Live Tables expectations, Great Expectations, dbt tests, or OHDSI Data Quality Dashboard (DQD), with results tracked across data refreshes.
  • Validating claims and EHR data products against source, vendor documentation, and published benchmarks, including record reconciliation, distribution and trend checks, and investigation and resolution of data anomalies.
Healthcare RWD — Claims and EHR
  • Administrative claims: deep, hands‑on working knowledge of medical and pharmacy claims from sources such as Optum Clinformatics, IQVIA PharMetrics, and Komodo, including enrollment/eligibility tables, revenue codes, place‑of‑service codes, inpatient vs. outpatient claim splitting, drug identification at NDC and GPI level, and known structural quirks of each source.
  • Complex EHR data: HL7 FHIR resources, Epic/Cerner/Truveta data models, clinical note schemas, problem list hierarchies, medication order and administration tables, lab result normalization, and vital sign time series — including semi‑structured and nested JSON/XML from EHR exports and FHIR APIs.
  • Healthcare terminology and ontology mapping: ICD‑10‑CM/PCS, ICD‑9‑CM, NDC, RxNorm, SNOMED CT, CPT‑4, HCPCS, LOINC, ATC — building, versioning, and governing code list repositories joined to raw tables to produce concept‑labeled analysis‑ready datasets.
  • OMOP CDM: transforming source claims and EHR data to OMOP v5.x including vocabulary loading (Athena), ETL specification documentation, and Achilles/DQD data quality checks.
  • Phenotyping and cohort construction: translating clinical study protocols into reproducible extraction logic — index date derivation, washout periods, time‑varying covariates, censoring — suitable for pharmacoepidemiology and HEOR studies.
  • Regulatory and privacy frameworks: HIPAA‑compliant data handling, de‑identification standards (Safe Harbor, Expert Determination), DUA compliance, and audit trail requirements for FDA‑grade RWE submissions.
AI Fluency
  • AI‑assisted software development: expert, daily use of AI coding assistants and agentic coding tools (e.g., Claude Code, GitHub Copilot, Cursor, Databricks Assistant) to write, refactor, debug, test, and document production code, with strong judgment on reviewing and validating AI‑generated code.
  • Building with frontier models: hands‑on experience integrating frontier AI model APIs (e.g., Anthropic, OpenAI, Google, Databricks Foundation Model APIs) into solutions that operate on data products, including prompt engineering, tool use/function calling, structured outputs, retrieval‑augmented generation (RAG), and agentic workflows.
  • Evaluating AI solutions: designing test sets and evaluation methods to measure the accuracy and reliability of AI outputs (e.g., natural‑language‑to‑SQL correctness, extraction accuracy), and implementing guardrails appropriate for patient‑level data.
  • Implementing AI‑powered data profiling and anomaly detection to automate quality checks across large ingestion runs and surface issues before they reach analytical teams.
  • Familiarity with MLOps tooling (MLflow, Databricks Model Serving, Vector Search) for versioning, deploying, and monitoring AI models integrated with data pipelines; experience applying NLP or LLMs to extract structured data from unstructured clinical notes a plus.
Minimum Qualification Requirements
  • Bachelor’s in Computer Science, Applied Mathematics, Engineering, Statistics, Information Technology, Bioinformatics.
  • Minimum 5 years' experience of hands‑on data engineering experience with a demonstrated focus on healthcare or life sciences RWD, including direct work with large administrative claims and/or EHR datasets.
  • Minimum of 1 year experience of hands‑on, production development on the Databricks platform using Python (PySpark), SQL, and Delta Lake, applied directly to real‑world healthcare data at scale.
  • Minimum of 1 year experience designing and implementing data quality and validation checks for large clinical datasets.
  • Minimum of 1 year of experience using AI coding assistants to develop production software, and experience building at least one solution using frontier AI model APIs.
Other Information/Additional Preferences
  • Experience with OMOP CDM transformation, vocabulary management (Athena), and federated analytics networks (OHDSI, PCORnet, Sentinel).
  • Master’s degree in Computer Science, Applied Mathematics, Engineering, Statistics, Information Technology, Bioinformatics.
  • Strong problem‑solving skills, attention to detail, and ability to work effectively in matrixed, cross‑functional teams with both technical and scientific partners.
  • Experience with distributed computing frameworks (Spark, Dask) for large‑scale RWD processing.
  • Experience building and deploying analytic applications using Dash, R Shiny, Streamlit, or similar frameworks.
  • Strong understanding of healthcare coding standards (ICD‑10, NDC, RxNorm, SNOMED CT, CPT, LOINC), claims and EHR data structures, and major RWD vendors and platforms (e.g., Merative MarketScan, Optum, IQVIA, Komodo, Truveta, HealthVerity).
  • Daily proficiency with Git‑based version control, including branching, pull requests, and code review.
  • Advanced SQL optimization skills on Databricks SQL / Spark SQL, including query plan analysis, query tuning, and compute cost management.
  • Demonstrated ability to build production‑grade ingestion pipelines for multi‑terabyte secondary healthcare datasets, including administrative claims and complex EHR data.
  • Experience with data pipeline observability and quality tooling (e.g., Great Expectations, Monte Carlo, Lakehouse Monitoring).
  • Familiarity with FDA RWE Framework guidance, EMA RWD guidance, and audit‑readiness requirements for observational studies.
  • Knowledge of Agile/Scrum methodologies and project management tools such as JIRA
  • Databricks certification (e.g., Data Engineer Associate or Professional, Generative AI Engineer Associate).
  • Experience with Model Context Protocol (MCP) or similar frameworks for connecting AI models to data and tools.
  • A portfolio of code (e.g., GitHub repositories) or deployed analytic applications demonstrating hands‑on engineering work.
  • Recruitment open across sectors: pharma/biotech data teams, RWD vendors (Truveta, Komodo, Optum, IQVIA, HealthVerity), cloud providers (AWS, Google, Microsoft), genomics platforms (Illumina, DNAnexus, Broad Institute), or academic health systems with large‑scale RWD programs.
Benefits
  • eligibility to participate in a company‑sponsored 401(k)
  • pension
  • vacation benefits
  • eligibility for medical, dental, vision and prescription drug benefits
  • flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts)
  • life insurance and death benefits
  • certain time off and leave of absence benefits
  • well‑being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities)

Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is

$136,500 - $200,200

Full‑time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance).

Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.

Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia (AMECA), Black Employees at Lilly (BE@Lilly), Chinese Culture Network (CCN), EnAble, Evolve, Lilly Indian Network (LIN), Organization of Latinx at Lilly (OLA), Pride (LGBTQ+ Allies), Veterans Leadership Network (VLN) and Women’s Initiative for Leading at Lilly (WILL).

##WeAreLilly

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Associate Director – RWD Data Engineering
Associate Director – RWD Data Engineering

Eli Lilly and Company • Indianapolis (IN)

On-site
USD 137,000 - 200,000
LillyDirect Pharmacy Technical Product Owner Patient Data
LillyDirect Pharmacy Technical Product Owner Patient Data

Biopharma Asia • Indianapolis (IN)

On-site
USD 129,000 - 189,000
401(k) plan
Pension benefits
Vacation benefits
+1
LillyDirect Pharmacy Technical Product Owner Patient Data
LillyDirect Pharmacy Technical Product Owner Patient Data

Eli Lilly and Company • United States

On-site
USD 129,000 - 189,000
Company bonus
401(k)
Comprehensive benefits
LillyDirect Pharmacy Data Engineer
LillyDirect Pharmacy Data Engineer

Eli Lilly and Company • United States

On-site
USD 66,000 - 189,000
Company bonus
401(k) and pension
Vacation benefits
+3
LillyDirect Pharmacy Technical Product Owner Patient Data
LillyDirect Pharmacy Technical Product Owner Patient Data

Eli Lilly and Company • Indianapolis (IN)

On-site
USD 129,000 - 189,000
401(k)
Pension
Vacation benefits
+1
Sr. Principal Data Scientist – Market Access
Sr. Principal Data Scientist – Market Access

Lilly • Indianapolis (IN)

On-site
USD 134,000 - 246,000
Company bonus
401(k) plan
Health benefits
LillyDirect Pharmacy Master Data Technical Lead
LillyDirect Pharmacy Master Data Technical Lead

BioSpace • Indianapolis (IN)

On-site
USD 129,000 - 231,000
LillyDirect Pharmacy Master Data Technical Lead
LillyDirect Pharmacy Master Data Technical Lead

Eli Lilly and Company • United States

On-site
USD 129,000 - 231,000
Company bonus
401(k) plan
Healthcare benefits
+1
Forward Deployment Lead, R&D Quality Strategy & Transformation
Forward Deployment Lead, R&D Quality Strategy & Transformation

Eli Lilly and Company • Indianapolis (IN)

On-site
USD 129,000 - 211,000
401(k)
Pension
Vacation benefits
+4
Principal AI/ML Engineer – Data Analytics & AI
Principal AI/ML Engineer – Data Analytics & AI

Eli Lilly and Company • Indianapolis (IN)

On-site
USD 69,000 - 165,000