Senior Data Platform Engineer: Scale PB Data & AI Pipelines

Matterworks, Inc.; Somerville (MA); Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

Health & Dental

Vision

Disability

Life Insurance

401k Match

Flexible Work

Commuter Benefits

Parking

Team Meals

Conferences

Job summary

Matterworks, Inc. is seeking a Senior Software Engineer to build the connective tissue of our data platform, enriching raw samples into datasets with biological context.

You will collaborate with ML researchers, scientists, and product teams on data pipelines, feature enrichment, and scalable data services. You will own pipelines, data contracts, and interfaces, while ensuring quality, cost efficiency, and secure data processing across the platform.

Qualifications

• Significant professional experience building production data systems and pipelines
• Proficient in Python and SQL for large-scale data processing
• Proficient in Kubernetes-native batch orchestration and modern data lake technologies (Argo Workflows, Metaflow, EKS, Glue, Athena, Apache Iceberg, Parquet, DuckDB, Terraform)
• Experience with LLM or agent components in production data paths, including eval loop and cost per record
• Daily use of AI coding tools and skepticism about production data correctness
• Owned work to running, validated systems, including backfills and debugging publish steps
• Comfort with messy scientific formats and toolchains (mzML, RDKit, ProteoWizard)
• Autonomy and eagerness to learn in an early-stage startup environment

Responsibilities

• Build and scale data contracts and pipelines for petabytes of data
• Scale data serving with cost-conscious storage, layout, and orchestration in Kubernetes
• Turn raw data into usable datasets with consistent schemas and metadata
• Automate quality gates to prevent regressions
• Own interfaces used by AI, chemistry, product, and agents
• Ensure operations meet SLAs with provenance and secure data processing

Skills

Data pipelines

Python

SQL

Kubernetes

Argo Workflows

Metaflow

Airflow

LLMs in production

Problem solving

Data quality

Tools

Argo Workflows

Metaflow

EKS

Glue

Athena

Apache Iceberg

Parquet

DuckDB

Terraform

Airflow

Dagster

Job description

Matterworks, Inc. is seeking a Senior Software Engineer to build the connective tissue of our data platform, enriching raw samples into datasets with biological context.

You will collaborate with ML researchers, scientists, and product teams on data pipelines, feature enrichment, and scalable data services. You will own pipelines, data contracts, and interfaces, while ensuring quality, cost efficiency, and secure data processing across the platform.