Senior Data Platform Engineer - Multi-Source Ingestion

Nir Yu; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

Nir Yu is seeking a senior data engineer to own the data platform from architecture to production, enabling concurrent intake from APIs, webhooks, files, and scraping. You will evolve current serial workflows into an event-driven, orchestrated platform with robust recovery and data lineage.

Responsibilities include building multi-source ingestion, organizing medallion data layers, and ensuring data quality with automated tests. Strong SQL, Python, and experience with dbt and GCP are required.

Qualifications

• Experience building multi-source production data platforms from zero to one, incl. architecture and deployment.
• Production experience with workflow orchestration tools (Airflow, Dagster, Prefect, Temporal) incl. retries and failure recovery.
• Hands-on production experience with dbt and Google Cloud Platform (GCP).
• Experience building event-driven systems with messaging or task queues and asynchronous processing.
• Strong SQL and PostgreSQL skills including data modeling and incremental loading.
• Knowledge of data contracts, raw and curated storage, retention policies, and automated quality checks.
• Strong Python skills with asynchronous HTTP and concurrent processing (asyncio, aiohttp, HTTPX).
• Web scraping experience with Scrapy, Playwright, Selenium, or Beautiful Soup, incl. concurrent scraping and rate limits.

Responsibilities

• Ingest data from multiple sources (APIs, webhooks, files, scraping) with concurrency and error handling.
• Orchestrate batch and event-driven workflows using modern tools (Airflow, Dagster, Prefect, Temporal).
• Implement Pub/Sub or Cloud Tasks-based intake with retries and replay safety.
• Preserve raw payloads in storage (GCS) before transformation with dbt.
• Define medallion layers (raw, standardized, curated) and ensure traceability to sources.
• Ensure data quality via automated validation and pipeline tests; monitor freshness and failures.
• Design and optimize PostgreSQL schemas, queries, and loading patterns for downstream products.
• Integrate provider enrichment and LLM-based structured extraction into the platform.

Skills

SQL

PostgreSQL

Python

Data modeling

Data contracts

Quality checks

Tools

dbt

Airflow

Dagster

Prefect

Temporal

GCP

BigQuery

Dataform

Scrapy

Playwright

Selenium

Beautiful Soup

Job description

Nir Yu is seeking a senior data engineer to own the data platform from architecture to production, enabling concurrent intake from APIs, webhooks, files, and scraping. You will evolve current serial workflows into an event-driven, orchestrated platform with robust recovery and data lineage.

Responsibilities include building multi-source ingestion, organizing medallion data layers, and ensuring data quality with automated tests. Strong SQL, Python, and experience with dbt and GCP are required.