Senior Data Engineer - Scientific AI Pipelines

Lila Sciences; San Francisco (CA); On-site; USD 144,000 - 240,000; Full time

Job summary

Lila Sciences is seeking a Data Engineer in San Francisco to build ETL pipelines and data models for its scientific data platform, collaborating with AI researchers and experimentalists to turn raw lab instrument outputs into validated datasets.

You will model heterogeneous data from bio, chemistry, and materials instruments, implement validation checks, and develop reusable analysis functions, while improving automation and observability across data flows.

Qualifications

• 2–6 years of experience in data engineering, bioinformatics, cheminformatics, or computational science.
• Strong Python skills, including typed, tested, production-quality code.
• Strong SQL skills, especially with Postgres or similar relational databases.
• Experience building ETL pipelines, data models, and reusable data transformations.
• Data science foundation, including statistics and pandas, NumPy, or similar tools.
• Experience translating noisy scientific measurements into accurate, validated datasets.
• Workflow orchestration experience, ideally Flyte, Airflow, Prefect, Dagster, or Nextflow.
• Active use of AI coding tools in day-to-day engineering work.

Responsibilities

• Design pipelines that turn raw lab output into analysis-ready scientific data.
• Model heterogeneous data from bio, chemistry, and materials instruments.
• Build validation checks, schema-evolution gates, and data quality workflows.
• Develop reusable analysis functions for scientific and AI research workflows.
• Improve automation and observability across instrument-to-result data flows.
• Build canonical datasets that scientists and AI researchers can trust.
• Use AI coding tools to accelerate pipeline development and team velocity.

Skills

Python

SQL

ETL pipelines

Data modeling

Statistics

Pandas/NumPy

Data quality

AI coding tools

Noisy data handling

Tools

Airflow

Dagster

Prefect

Flyte

Nextflow

Parquet

Iceberg

DuckDB

Polars

Ibis

Job description

Lila Sciences is seeking a Data Engineer in San Francisco to build ETL pipelines and data models for its scientific data platform, collaborating with AI researchers and experimentalists to turn raw lab instrument outputs into validated datasets.

You will model heterogeneous data from bio, chemistry, and materials instruments, implement validation checks, and develop reusable analysis functions, while improving automation and observability across data flows.