Data Engineer — Life Sciences Data Pipelines

BioSpace; Indianapolis (IN); On-site; USD 65,000 - 158,000; Full time

Benefits offered by this job

Bonus program

Comprehensive benefits

401(k)

Job summary

Lilly is seeking a Data Engineer to design, build, and maintain robust data pipelines and architecture to support scientific and business initiatives. The role emphasizes Databricks, PySpark, SQL, Delta Lake, and secure data handling within the CE boundary.

Responsibilities include ELT/ETL workflow implementation, CI/CD for data artifacts, and collaboration with data scientists and stakeholders to ensure scalable, compliant data solutions for life-changing medicines.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
• 5+ years of experience in data engineering or ETL development.
• Proficiency in SQL and at least one programming language (Python/Java).
• Experience with cloud data platforms (Databricks, AWS, Azure, GCP) and data services (S3, Redshift, Snowflake, ADLS, BigQuery).
• Proficiency with Git-based CI/CD workflows for versioned data and pipelines.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
• Build Databricks pipelines with Bronze-Silver-Gold medallion architecture inside CE trust boundary.
• Evaluate Databricks capabilities and integration patterns (Unity Catalog, Delta Lake, Workflows, serverless compute).
• Implement ELT/ETL workflows using Databricks Workflows, Auto Loader, Structured Streaming, and Delta Live Tables.
• Create CI/CD pipelines (GitHub Actions) for CE data, contracts, policies, and artifacts.
• Automate data ingestion and product creation to reduce manual pipeline maintenance.

Skills

SQL

Python/Java

Databricks

Cloud platforms

Git CI/CD

Education

Bachelor's degree in CS/Engineering

Tools

Databricks

AWS

Azure

GCP

Job description

Lilly is seeking a Data Engineer to design, build, and maintain robust data pipelines and architecture to support scientific and business initiatives. The role emphasizes Databricks, PySpark, SQL, Delta Lake, and secure data handling within the CE boundary.

Responsibilities include ELT/ETL workflow implementation, CI/CD for data artifacts, and collaboration with data scientists and stakeholders to ensure scalable, compliant data solutions for life-changing medicines.