Senior Data Engineer

Hack The Box LTD; Virginia, New York, Town of Florida (MN, NY, NY); On-site; USD 140,000 - 160,000; Full time

Benefits offered by this job

Medical, Dental & Vision coverage

401K with employer match

Life and AD&D insurance

Disability insurance

Paid parental leave

Home Office Allowance

Training budget

State-of-the-art equipment

Full Hack The Box lab access

Job summary

Hack The Box LTD is seeking a Senior Data Engineer to own and evolve our data pipelines on a GCP-based stack, building new pipelines and hardening existing ones to deliver clean, trustworthy data for analytics, ML training, and online inference.

You will design ELT/ETL processes on BigQuery and ClickHouse, while building real-time pipelines with Pub/Sub and Kafka using Dataflow, and orchestrating workflows with Airflow.

Qualifications

• Strong data modelling and warehouse architecture skills.
• Hands-on experience with GCP data services—BigQuery is a must; Pub/Sub, Dataflow, Bigtable, Cloud Composer are strong pluses.
• Production experience with streaming pipelines on Dataflow/Beam, Flink, or Spark Structured Streaming.
• Solid SQL and strong Python — production-quality code.
• Experience with ClickHouse or another columnar OLAP engine in production.
• Workflow orchestration experience with Airflow (or Prefect/Dagster).
• Comfortable with dbt or equivalent transformation frameworks.
• Experience migrating off legacy warehouses (Snowflake, Redshift, Synapse).
• Working knowledge of ML in production — feature engineering and retraining.

Responsibilities

• Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery, Bigtable, and ClickHouse.
• Help drive the migration off Snowflake onto the GCP-native stack.
• Own the orchestration layer in Airflow, including SLAs, retries, and data quality gates.
• Model data for analytics and for ML — including feature pipelines for training and online inference.
• Partner with ML engineers on feature stores, drift monitoring, and retraining workflows.
• Capture requirements from stakeholders and translate them into data products.
• Continuously improve data quality, observability, and cost efficiency.
• Identify new data sources and integrate them cleanly.

Skills

Data modelling

GCP data services

SQL

Python

Airflow

dbt

Docker & Kubernetes

Data quality gates

Snowflake migration

Machine learning feature pipelines

Tools

BigQuery

Pub/Sub

Dataflow

Bigtable

Cloud Composer

Kafka

ClickHouse

Job description

The core mission of the Senior Data Engineer:

You will own and evolve our data pipelines on GCP — building new ones, hardening existing ones, improving data quality, and making clean, trustworthy data available across the organisation. You'll work end-to-end on streaming and batch pipelines, from CDC and event ingestion through transformation, serving, and the feature layer that powers our ML and AI products.

Your day-to-day will include designing ELT/ETL processes on BigQuery and ClickHouse, building real-time pipelines on Pub/Sub and Kafka with Dataflow (and where it fits, Flink/Spark), orchestrating workflows with Airflow, and ensuring data is properly cleaned, modelled, and served for analytics, ML training, and online inference. You'll partner with ML engineers on feature pipelines, monitoring data drift, and keeping models well‑fed and retrained as needed. You'll consume and build REST APIs, integrate with third‑party SaaS sources, and treat infrastructure as code.

Location & Work Mode:

• US
• Fully Remote

The fellowship you’ll be joining:

You will be part of the Data, Analytics & AI team, collaborating closely with Infrastructure, Software Engineering, Product, and ML/AI engineers. We’re in the middle of a GCP‑native modernisation — migrating away from Snowflake toward BigQuery, Bigtable, Pub/Sub, and Dataflow — so we’re looking for someone who is opinionated about clean architecture, allergic to over‑engineering, and comfortable owning systems end‑to‑end. If retiring a legacy warehouse and standing up its replacement sounds like a good time, you’ll fit right in.

Technology tools you’ll be using:

• Cloud & warehouse: GCP, BigQuery, Bigtable, Cloud Storage
• Streaming & messaging: Pub/Sub, Kafka
• Processing: Dataflow (Apache Beam), with Flink/Spark where appropriate
• Orchestration: Airflow (Cloud Composer)
• Analytical store: ClickHouse
• Languages: Python, SQL
• Modelling & quality: dbt, data quality gates
• Containers & CI/CD: Docker, Kubernetes, GitHub Actions or equivalent
• Legacy being retired: Snowflake

The adventures that await you after becoming Senior Data Engineer:

• Design and build batch and streaming pipelines on Dataflow, Pub/Sub, and Kafka feeding BigQuery, Bigtable, and ClickHouse
• Help drive the migration off Snowflake onto our GCP‑native stack — and retire shadow pipelines along the way
• Own the orchestration layer in Airflow, including SLAs, retries, and data quality gates
• Model data for analytics and for ML — including feature pipelines that serve both training and low‑latency online inference
• Partner with ML engineers on feature stores, drift monitoring, and retraining workflows
• Capture requirements from stakeholders and translate them into pragmatic, well‑scoped data products
• Continuously improve data quality, reliability, observability, and cost efficiency
• Identify new data sources worth acquiring and integrate them cleanly

Skills, knowledge, and experience points required:

• Strong data modelling and warehouse architecture skills (dimensional modelling, event‑driven, lakehouse patterns)
• Hands‑on experience with GCP data services — BigQuery is a must; Pub/Sub, Dataflow, Bigtable, Cloud Composer are strong pluses
• Production experience with streaming pipelines on Dataflow/Beam, Flink, or Spark Structured Streaming, ingesting from Kafka and/or Pub/Sub
• Solid SQL and strong Python — you write production‑quality code, not just notebooks
• Experience with ClickHouse or another columnar OLAP engine in production
• Workflow orchestration experience with Airflow (or Prefect/Dagster)
• Comfortable with dbt or equivalent transformation frameworks
• Experience migrating off legacy warehouses (Snowflake, Redshift, Synapse) onto cloud‑native stacks is a plus
• Working knowledge of ML in production — feature engineering, feature stores, model deployment, drift monitoring, retraining
• Docker & Kubernetes experience
• CI/CD mindset, infrastructure‑as‑code sensibility, and a bias for simple, observable systems
• Bonus: CDC tooling (Datastream, Debezium), Vertex AI or Feature Store

The gems you’ll be enjoying as Senior Data Engineer:

• Compensation: $140,000 - 160,000
• Medical, Dental & Vision (employee coverage 100% paid for by Hack The Box)
• 401K with employer match
• Employer‑paid Life and AD&D insurance
• Supplemental Life Insurance
• Short‑term and Long‑term Disability
• Healthcare and Dependent Care FSA
• Paid parental leave
• 25 annual leave days
• Home Office Allowance
• Dedicated budget for training and professional development, participation in conferences
• State‑of‑the‑art equipment
• Full access to the Hack The Box lab offerings; so you can learn how to hack

At Hack The Box, we are committed to fostering a diverse, inclusive, and equitable workplace. We believe that diversity enriches our performance, services, and the communities we serve. As such, we ensure all job applications are considered solely based on merit, skills, and qualifications. We do not discriminate on grounds of race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status. We are dedicated to providing a fair and respectful work environment that reflects our values.