Role overview
Own the architecture, implementation, and production operation of a data platform that consolidates information from APIs, webhooks, files, and web scraping. You’ll evolve existing serial workflows into concurrent, event-driven intake and orchestrated batch processing, delivering traceable, dependable records to downstream products.
Responsibilities
- Build ingestion across multiple providers and formats, handling concurrency, rate limits, retries, and failures.
- Orchestrate scheduled jobs and historical backfills with a workflow engine; implement event-driven processing with queues, dead-letter handling, and controlled replay.
- Preserve original source payloads in cloud storage and transform them into raw, standardized, and curated data layers.
- Design idempotent updates and safe recovery paths to support reprocessing without duplicates or corruption.
- Create data contracts, automated validation, pipeline tests, and monitoring for freshness and failures.
- Design and optimize PostgreSQL schemas, queries, and incremental loading for reliable downstream delivery.
- Integrate provider enrichment and structured extraction from text using language models.
Requirements
- Experience building production data platforms from initial architecture through deployment and ongoing operations.
- Production experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal, including retries and recovery.
- Hands-on experience with dbt and Google Cloud Platform.
- Experience with event-driven systems, messaging or task queues, asynchronous processing, and failure recovery.
- Strong SQL and PostgreSQL skills, including modeling, query tuning, and incremental loads.
- Strong Python skills, including asynchronous HTTP and concurrent processing.
- Practical web scraping experience, including concurrency, rate limits, and fault handling.
- Familiarity with data contracts, storage retention, and automated quality checks.
Nice to have
- Experience in a startup or small engineering team with broad ownership and shifting priorities.
- Entity resolution skills, such as matching company or person records and reconciling duplicates.
- Experience with language-model extraction or company and people data enrichment.
- Familiarity with BigQuery and Dataform.