Hybrid Data Engineer: AI‑Driven Drug Discovery

Initial Therapeutics, Inc.; San Francisco (CA); On-site; USD 158,000 - 231,000; Full time

Benefits offered by this job

401(k) plan

Pension

Healthcare benefits

Flexible benefits

Well-being programs

Job summary

Lilly is seeking a Data Engineer for our Silicon Valley hub with 5+ years of experience to build scalable data platforms powering AI-driven research and discovery. You will ingest, transform, and deliver chemical, biological, and experimental data for machine learning workflows across teams.

You will design robust data pipelines, ensure data quality, and enable reproducible experiments with versioned datasets, collaborating with AI scientists and lab researchers in a flexible hybrid model.

Qualifications

• Bachelor’s degree in Computer Science, Data Science, Engineering, Mathematics, or a related technical field.
• 5+ years of data engineering experience building and operating production data systems.

Responsibilities

• Engineer datasets in large language environments with efficient formats (Parquet, Zarr, Arrow) and fast delivery for GPU clusters.
• Design, develop, and maintain scalable data pipelines for data analytics, reporting, and ML initiatives.
• Ensure seamless data flow between systems, optimizing transfer and transformation processes for performance and scalability.
• Build ingestion paths from automated lab data to models, reducing time to learn from experiments.
• Own data correctness, complete joins, and dataset validation before training.
• Implement dataset versioning, lineage, and reproducibility across training runs.
• Collaborate with lab, instrument, and external teams to prevent upstream changes from breaking downstream training.

Skills

Python

SQL

Data pipelines

Spark

Airflow

Dagster

Cloud platforms

Kafka

MQTT

AMQP

CI/CD

Git

Parquet

Data modeling

Education

Bachelor’s degree in Computer Science, Data Science, Engineering, Mathematics, or related technical field

Tools

Postgres

Job description

Lilly is seeking a Data Engineer for our Silicon Valley hub with 5+ years of experience to build scalable data platforms powering AI-driven research and discovery. You will ingest, transform, and deliver chemical, biological, and experimental data for machine learning workflows across teams.

You will design robust data pipelines, ensure data quality, and enable reproducible experiments with versioned datasets, collaborating with AI scientists and lab researchers in a flexible hybrid model.