Senior Data Platform Engineer - Real-Time & Streaming

Whatnot; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Health Insurance

Work From Home Support

Home office setup allowance

Cell phone & internet allowance

Childcare allowance

Parental leave

401k matching

Dogfooding budget

Job summary

Whatnot is seeking a senior Software Engineer, Data Foundations, to shape the core data platform that powers our live commerce experiences. You will define technical direction, lead development of streaming and orchestration platforms, and own platform reliability and SLOs at scale.

You will collaborate with product, platform, ML, and data teams to deliver real-time analytics, stateful data pipelines, and scalable data architectures across multi-tenant environments.

Qualifications

• Ability to write well-tested, API-driven applications and services that run at scale in production.
• Track record of building scalable, high-performance distributed systems.
• Strong experience with streaming data systems (Kafka/Flink/Pulsar) and/or orchestration platforms (Airflow/Dagster/Temporal).
• Hands-on experience designing and operating stateful, real-time data pipelines at scale (checkpointing, state management, watermarking).
• Experience evaluating architectural tradeoffs in data processing/orchestration systems.
• Experience building multi-tenant data infrastructure for multiple teams or production use cases.
• Expertise in managing cloud data stores (Snowflake/BigQuery/Redshift) and data lake architectures (AWS S3).
• Strong experience supporting operational systems: monitoring, alerting, incident management.
• Familiarity with DevOps/SRE and CI/CD tooling, including GitHub Actions.
• Understanding of data governance and compliance (access controls, retention, auditability).
• Experience managing cloud technologies (AWS/Google Cloud) and infrastructure-as-code (Terraform).
• Proficiency in Python and core CS concepts; strong problem solving and communication.

Responsibilities

• Define technical direction and lead development of data platform systems.
• Lead buildout and evolution of data orchestration and streaming platforms.
• Design scalable, multi-tenant data infrastructure and platform layer hardware backplane.
• Own scalability, performance, reliability, SLOs, and operations for data processing systems.
• Build real-time analytics and ML infrastructure capabilities.
• Collaborate with product, platform, ML, and data teams to realize architecture.
• Shape enterprise architecture across orchestration, stream processing, and analytics.
• Apply DevOps/SRE practices to keep systems reliable at scale.
• Define and enforce data governance and compliance standards across data lake and downstream systems.

Skills

APIs at scale

Distributed systems

Streaming data (Kafka/Flink/Pulsar)

Orchestration platforms (Airflow/Dagst

Tools

Kafka

Flink

Pulsar

Airflow

Dagster

Temporal

Snowflake

BigQuery

Redshift

AWS S3

GitHub Actions

Terraform

Job description

Whatnot is seeking a senior Software Engineer, Data Foundations, to shape the core data platform that powers our live commerce experiences. You will define technical direction, lead development of streaming and orchestration platforms, and own platform reliability and SLOs at scale.

You will collaborate with product, platform, ML, and data teams to deliver real-time analytics, stateful data pipelines, and scalable data architectures across multi-tenant environments.