Senior Data Engineer - Remote, GCP Pipelines & ML Features

Hack The Box; United States; On-site; USD 140,000 - 160,000; Full time

Benefits offered by this job

Medical, Dental & Vision

401K with employer match

Life and AD&D Insurance

Disability insurance

Paid parental leave

25 annual leave days

Training budget for conferences

Full access to Hack The Box labs

Job summary

Hack The Box is seeking a Senior Data Engineer to own and evolve our data pipelines on GCP, building ELT/ETL, streaming and batch capabilities to support analytics, ML training, and online inference.

You will partner with ML engineers, implement Airflow, and drive the migration from Snowflake to a cloud-native stack (BigQuery, Bigtable, Dataflow), ensuring data quality and observability across the platform.

Qualifications

• Strong data modelling and warehouse architecture understanding.
• Experience with GCP data services (BigQuery, Pub/Sub, Dataflow).
• Production experience with streaming pipelines (Dataflow/Beam, Kafka/Pub/Sub).
• Solid SQL and Python; production-quality code.
• Experience with ClickHouse or similar OLAP.
• Airflow or similar orchestration experience.
• Familiarity with dbt or equivalent transformation frameworks.
• Experience migrating from Snowflake/Redshift to cloud-native stacks is a plus.

Responsibilities

• Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery/Bigtable/ClickHouse.
• Lead the Snowflake migration to a GCP-native stack and retire shadow pipelines.
• Own the orchestration layer in Airflow with SLAs, retries, and data quality gates.
• Model data for analytics and ML feature pipelines for training and online inference.
• Collaborate with ML engineers on feature stores, drift monitoring, and retraining workflows.
• Translate stakeholder requirements into pragmatic, well-scoped data products.
• Continuously improve data quality, reliability, observability, and cost efficiency.
• Identify new data sources and integrate them cleanly.

Skills

Strong data modelling

SQL proficiency

Python programming

ML collaboration

Tools

BigQuery

Pub/Sub

Dataflow

ClickHouse

Airflow

Kafka

Docker

Kubernetes

dbt

Job description

Hack The Box is seeking a Senior Data Engineer to own and evolve our data pipelines on GCP, building ELT/ETL, streaming and batch capabilities to support analytics, ML training, and online inference.

You will partner with ML engineers, implement Airflow, and drive the migration from Snowflake to a cloud-native stack (BigQuery, Bigtable, Dataflow), ensuring data quality and observability across the platform.