Senior Data Engineer: End-to-End Platform Ownership

IllTip; United States; Remote; USD 150,000 - 210,000; Full time

Job summary

IllTip seeks a Senior Data Platform Engineer to own the data platform from ingestion architecture to production operations. The role emphasizes broad ownership of design and day-to-day reliability for trusted, well-tested records that power data products.

You will design and operate a platform handling concurrent ingestion, event-driven processing, and orchestrated batch workflows, with data ingested from APIs, webhooks, files, and web scraping.

Qualifications

• 5+ years building and operating production data platforms; 8+ years is a plus.
• Production experience with workflow orchestration (Airflow preferred) including dependencies, retries, and recovery.
• Hands-on production experience with dbt and Google Cloud Platform.
• Strong Python (asyncio, aiohttp, or HTTPX for concurrent HTTP) and strong SQL and PostgreSQL skills.
• Experience with messaging or task queues, data contracts, retention policies, automated validation, and monitoring.
• Web scraping (Scrapy, Playwright, Selenium, Beautiful Soup), entity resolution, or LLM-based extraction experience is a plus.

Responsibilities

• Design and build a platform that handles concurrent ingestion, event-driven processing, and orchestrated batch workflows.
• Ingest data from provider APIs, webhooks, files, and web scraping, with rate limiting and strong failure handling.
• Run scheduled workflows and historical backfills in an orchestrator such as Airflow, Dagster, or Prefect.
• Build queue-based event systems with retries, dead-letter handling, and controlled replay.
• Store raw data in cloud storage and transform it with dbt into raw, standardized, and curated layers; design and tune PostgreSQL schemas and loading patterns.
• Implement idempotent loads, data contracts, validation, pipeline tests, freshness monitoring, and failure alerts.

Skills

Python (asyncio)

SQL / PostgreSQL

Data pipeline design

Distributed systems

Cloud platforms

Tools

Airflow

Dagster

Prefect

dbt

Google Cloud Platform

Job description

IllTip seeks a Senior Data Platform Engineer to own the data platform from ingestion architecture to production operations. The role emphasizes broad ownership of design and day-to-day reliability for trusted, well-tested records that power data products.

You will design and operate a platform handling concurrent ingestion, event-driven processing, and orchestrated batch workflows, with data ingested from APIs, webhooks, files, and web scraping.