Data Engineer — Batch & Streaming Pipelines

Hivemind

New York (NY)

Remote

USD 120,000 - 170,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health, dental, vision insurance
401(k) with 4% company match
Equity option plan
Home office stipend
DoorDash meals
Wellness stipend

Job summary

Rain builds payments and stablecoin infrastructure. As a Data Engineer, you will design and maintain batch and streaming pipelines, ingested data from databases, files, APIs, and onchain sources, and ensure accuracy in our warehouse.

You will work with Dagster, dbt, and cloud warehouses (BigQuery, Snowflake, ClickHouse) to drive data quality, enable ML features, and partner with analytics engineers and risk teams.

Qualifications

  • 3+ years building production data pipelines end-to-end.
  • Experience designing data solutions at scale.
  • Strong SQL and Python or Rust.
  • Experience with a cloud data warehouse such as BigQuery, ClickHouse, or Snowflake.
  • Streaming with Pub/Sub, Kafka, Dataflow/Beam, or Flink.
  • Batch with dbt and Dagster or Airflow.
  • Production data quality testing, monitoring, and reconciliation.
  • Ability to debug across source, pipeline, warehouse and report.
  • Ability to use AI tools to learn and work faster while checking output.

Responsibilities

  • Build scalable batch and streaming pipelines, choosing the right approach for latency, cost, and correctness.
  • Process events near real time, handling deduplication, windowing, and late or out-of-order events.
  • Own batch ingestion from databases (CDC), files (SFTP and GCS), APIs, and onchain sources.
  • Build incremental dbt models that handle late data, duplicates, CDC deletes, and backfills; reconcile streaming and batch outputs.
  • Orchestrate pipelines in Dagster with sensors, schedules, and alerts.
  • Add freshness, volume, and reconciliation checks to every source, and fix root causes.
  • Work with backend pods on event and table contracts to prevent downstream breakage.
  • Partner with Analytics Engineers, Data Scientists, and Risk on their data needs, including ML features.

Skills

SQL
Python
Rust
Data modeling
Problem solving
Communication

Tools

BigQuery
ClickHouse
Snowflake
Pub/Sub
Kafka
Dataflow/Beam
Flink
dbt
Dagster
Airflow
Airbyte

Job description

Rain builds payments and stablecoin infrastructure. As a Data Engineer, you will design and maintain batch and streaming pipelines, ingested data from databases, files, APIs, and onchain sources, and ensure accuracy in our warehouse.

You will work with Dagster, dbt, and cloud warehouses (BigQuery, Snowflake, ClickHouse) to drive data quality, enable ML features, and partner with analytics engineers and risk teams.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer (Batch & Streaming) - Hybrid + Equity
Data Engineer (Batch & Streaming) - Hybrid + Equity

Hivemind • United States

Hybrid
USD 150,000 - 190,000
Unlimited vacation
Flexible work
Health/dental/vision 95% covered
+4
Data Engineer - Real-Time Pipelines & Quality at Scale
Data Engineer - Real-Time Pipelines & Quality at Scale

raincards.xyz • San Juan (PR), New York (NY)

Remote
USD 120,000 - 160,000
Unlimited vacation with 10 days min
Flexible work location (home/office)
Home office stipend
+4
Data Engineer: Build Real-Time Pipelines & Trustworthy Data
Data Engineer: Build Real-Time Pipelines & Trustworthy Data

Rain • San Juan (PR)

Hybrid
USD 120,000 - 180,000
Unlimited vacation with 10 days min.
Flexible working (home/office) with a设
401(k) with company match
+2
Data Engineer: Real-Time & Batch Pipelines
Data Engineer: Real-Time & Batch Pipelines

Solana Teams • San Juan (PR)

Hybrid
USD 120,000 - 190,000
Unlimited vacation with minimum 10‑day
Home office stipend
Health/dental/vision coverage (US)
+3
Data Engineer - Real-Time & Batch Pipelines (Hybrid)
Data Engineer - Real-Time & Batch Pipelines (Hybrid)

rain • Puerto Rico

Remote
USD 110,000 - 190,000
Flexible time off
Remote-friendly
Home office stipend
+3
Data Engineer — Real-Time Pipelines & Data Quality
Data Engineer — Real-Time Pipelines & Data Quality

Rain • United States

Hybrid
USD 120,000 - 180,000
Flexible time off
Work from home
Office stipend
+3
Data Engineer: Real-Time Pipelines, AI QA, Remote
Data Engineer: Real-Time Pipelines, AI QA, Remote

Rain • New York (NY)

Hybrid
USD 120,000 - 190,000
Unlimited vacation (min 10 days)
Flexible work location
Home workspace stipend
+6
Data Platform Engineer – Batch & Streaming, Equity
Data Platform Engineer – Batch & Streaming, Equity

raincards.xyz • San Juan (PR), New York (NY)

Hybrid
USD 150,000 - 210,000
Unlimited vacation
Flexible work
Home workspace stipend
+5
Data Platform Engineer: Cloud, Streaming & ML Pipelines
Data Platform Engineer: Cloud, Streaming & ML Pipelines

Rain • San Juan (PR)

Hybrid
USD 140,000 - 180,000
Unlimited vacation
Flexible working (work from home or in
Home office stipend
+6
Data Engineer - Real-Time Pipelines & Scalable Data
Data Engineer - Real-Time Pipelines & Scalable Data

Rain Xyz • United States

Remote
USD 120,000 - 160,000