Senior Data Engineer: Scalable Ingestion & Data Quality

Fundable LLC; United States; Remoto; USD 120.000 - 160.000; Tempo pieno

Descrizione del lavoro

Fundable LLC in the United States is seeking a data engineer to build concurrent ingestion for provider APIs, webhooks, files, and scraping with robust rate limits and failure handling. You will orchestrate scheduled workflows with Airflow, Dagster, or Prefect, implement event-driven intake with Pub/Sub or Cloud Tasks, and ensure data quality through validation and testing.

Join a small engineering team focused on reliable data infrastructure and scalable PostgreSQL schemas for downstream

Competenze

• Experience designing reliable data ingestion pipelines with multiple data sources.
• Ability to implement idempotent processing and safe retries for backfills.
• Familiarity with data contracts, automated validation, and pipeline tests.

Mansioni

• Build concurrent ingestion for provider APIs, webhooks, files, and scraping with rate limits and failure handling.
• Orchestrate scheduled workflows and historical backfills using Airflow, Dagster, or Prefect.
• Implement event-driven intake with Pub/Sub or Cloud Tasks including retries and dead-letter queues.
• Design and optimize PostgreSQL schemas and queries for downstream products.

Conoscenze

Airflow

Dagster

Prefect

Pub/Sub

Cloud Tasks

Backfills

Retry strategies

Strumenti

dbt

PostgreSQL

Google Cloud Storage

Descrizione del lavoro

Fundable LLC in the United States is seeking a data engineer to build concurrent ingestion for provider APIs, webhooks, files, and scraping with robust rate limits and failure handling. You will orchestrate scheduled workflows with Airflow, Dagster, or Prefect, implement event-driven intake with Pub/Sub or Cloud Tasks, and ensure data quality through validation and testing.

Join a small engineering team focused on reliable data infrastructure and scalable PostgreSQL schemas for downstream