Remote Data Engineer: GCP Streaming & ML-Ready Data

Hack The Box; Town of Florida (NY); On-site; USD 140,000 - 160,000; Full time

Job summary

Hack The Box is seeking a Senior Data Engineer to design and operate end-to-end data pipelines on GCP, enabling analytics, ML training, and online inference. You will drive the migration from Snowflake to a GCP-native stack and own Airflow orchestration with robust data quality gates.

Responsibilities include building ELT/ETL on BigQuery and ClickHouse, streaming pipelines with Pub/Sub and Kafka, and collaborating with ML engineers on feature pipelines.

Qualifications

• Strong data modelling and warehouse architecture skills (dimensional modelling, event-driven, lakehouse patterns).
• Hands-on experience with GCP data services — BigQuery is a must; Pub/Sub, Dataflow, Bigtable, Cloud Composer are strong pluses.
• Production experience with streaming pipelines on Dataflow/Beam, Flink, or Spark, ingesting from Kafka and/or Pub/Sub.

Responsibilities

• Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery, Bigtable, and ClickHouse.
• Help drive the migration off Snowflake onto our GCP-native stack and retire shadow pipelines.
• Own the orchestration layer in Airflow, including SLAs, retries, and data quality gates.

Skills

GCP data services

BigQuery

Pub/Sub

Kafka

Dataflow

Airflow

Python

SQL

ClickHouse

Docker

Kubernetes

dbt

CI/CD

Infrastructure as code

Snowflake migration

Tools

BigQuery

Pub/Sub

Kafka

Dataflow (Beam)

Airflow (Cloud Composer)

ClickHouse

Snowflake

Docker

Kubernetes

Job description

Hack The Box is seeking a Senior Data Engineer to design and operate end-to-end data pipelines on GCP, enabling analytics, ML training, and online inference. You will drive the migration from Snowflake to a GCP-native stack and own Airflow orchestration with robust data quality gates.

Responsibilities include building ELT/ETL on BigQuery and ClickHouse, streaming pipelines with Pub/Sub and Kafka, and collaborating with ML engineers on feature pipelines.