Senior Data Engineer: GCP Pipelines for ML Features

Hack The Box; New Jersey; On-site; USD 140,000 - 160,000; Full time

Benefits offered by this job

Medical, Dental & Vision

401(k) with employer match

Life and AD&D Insurance

Paid parental leave

25 annual leave days

Home Office Allowance

Training & conference budget

Equipment & lab access

Job summary

Hack The Box is seeking a data engineer to own and evolve data pipelines on GCP, improving data quality and reliability across the organisation. You will work end‑to‑end on streaming and batch pipelines, from ingestion to the feature layer powering ML/AI products.

You’ll join the Data, Analytics & AI team, collaborating with Infrastructure, Software Eng, Product, and ML/AI engineers. Strong SQL, Python, and GCP experience are essential.

Qualifications

• Strong data modelling and warehouse architecture skills (dimensional modelling, event‑driven, lakehouse patterns).
• Hands‑on experience with GCP data services — BigQuery is a must; Pub/Sub, Dataflow, Bigtable, Cloud Composer are strong pluses.
• Production experience with streaming pipelines on Dataflow/Beam, Flink, or Spark Structured Streaming, ingesting from Kafka and/or Pub/Sub.
• Solid SQL and strong Python — you write production‑quality code, not just notebooks.
• Experience with ClickHouse or another columnar OLAP engine in production.
• Workflow orchestration experience with Airflow (or Prefect/Dagster).
• Comfortable with dbt or equivalent transformation frameworks.
• Experience migrating off legacy warehouses (Snowflake, Redshift, Synapse) onto cloud‑native stacks is a plus.
• Working knowledge of ML in production — feature engineering, feature stores, model deployment, drift monitoring, retraining.
• Docker & Kubernetes experience.
• CI/CD mindset, infrastructure‑as‑code sensibility, and a bias for simple, observable systems.
• Bonus: CDC tooling (Datastream, Debezium), Vertex AI / Feature Store.

Responsibilities

• Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery, Bigtable, and ClickHouse.
• Help drive the migration off Snowflake onto our GCP‑native stack — and retire shadow pipelines along the way.
• Own the orchestration layer in Airflow, including SLAs, retries, and data quality gates.
• Model data for analytics and for ML — including feature pipelines that serve both training and low‑latency online inference.
• Partner with ML engineers on feature stores, drift monitoring, and retraining workflows.
• Capture requirements from stakeholders and translate them into pragmatic, well‑scoped data products.
• Continuously improve data quality, reliability, observability, and cost efficiency.
• Identify new data sources worth acquiring and integrate them cleanly.

Skills

GCP data services

BigQuery

Pub/Sub

Dataflow/Beam

Flink / Spark

Airflow

ClickHouse

Python

SQL

dbt

Docker

Kubernetes

CI/CD

Tools

GitHub Actions

Cloud Composer

Job description

Hack The Box is seeking a data engineer to own and evolve data pipelines on GCP, improving data quality and reliability across the organisation. You will work end‑to‑end on streaming and batch pipelines, from ingestion to the feature layer powering ML/AI products.

You’ll join the Data, Analytics & AI team, collaborating with Infrastructure, Software Eng, Product, and ML/AI engineers. Strong SQL, Python, and GCP experience are essential.