Senior Data Engineer: Real-Time Data Pipelines on GCP

Hack The Box; Town of Florida (NY); On-site; USD 140,000 - 160,000; Full time

Benefits offered by this job

Medical, Dental & Vision

401K with match

Life Insurance

Paid parental leave

Home Office Allowance

Job summary

Hack The Box is seeking a Senior Data Engineer to own and evolve our data pipelines on GCP, building new ones, hardening existing ones, and ensuring clean, trustworthy data for analytics, ML, and online inference. You will design ELT/ETL processes on BigQuery and ClickHouse, and build real-time pipelines with Pub/Sub and Kafka.

You will collaborate across teams, mentor junior engineers, and help migrate from Snowflake to our cloud-native stack, maintaining data quality, observability, and cost

Qualifications

• Strong data modelling and warehouse architecture skills.
• Hands-on experience with GCP data services (BigQuery essential).
• Production experience with streaming pipelines (Dataflow, Beam).
• Solid SQL and Python for production code.
• Experience with ClickHouse or similar OLAP engines.
• Experience with Airflow or similar orchestration tools.
• Familiarity with dbt or equivalent transformation frameworks.

Responsibilities

• Design and build batch and streaming pipelines on Dataflow and Kafka.
• Migrate from Snowflake to a cloud-native stack and retire legacy pipelines.
• Own the orchestration layer in Airflow with SLAs and retries.
• Model data for analytics and ML feature pipelines.
• Collaborate with ML engineers on feature stores and drift monitoring.
• Capture stakeholder requirements and translate into data products.
• Improve data quality, observability, and cost efficiency.

Skills

Data modelling

GCP data services

Python

SQL

Airflow

dbt

Docker/Kubernetes

Tools

BigQuery

Bigtable

Pub/Sub

Dataflow

ClickHouse

Job description

Hack The Box is seeking a Senior Data Engineer to own and evolve our data pipelines on GCP, building new ones, hardening existing ones, and ensuring clean, trustworthy data for analytics, ML, and online inference. You will design ELT/ETL processes on BigQuery and ClickHouse, and build real-time pipelines with Pub/Sub and Kafka.

You will collaborate across teams, mentor junior engineers, and help migrate from Snowflake to our cloud-native stack, maintaining data quality, observability, and cost