Senior Data Engineer - 100TB/Day AI Pipelines

The Hotels Network; Oost-Vlaanderen; Hybrid; EUR 90,000 - 130,000; Full time

Benefits offered by this job

Flexible work from home

Health insurance

Pension

Lunch vouchers

Mobility budget

Job summary

Lighthouse is looking for a Senior Data Engineer to own the end-to-end lifecycle of its large-scale ingestion and transformation engine. You will design, develop, scale, and maintain data pipelines handling massive data streams around 100 TB per day.

You will work with Python, dbt, Airflow on Google Cloud Platform, BigQuery, Spanner, Pub/Sub, Kubernetes, and Claude, collaborating with product, engineering, support, and business teams to deliver high-fidelity data products.

Qualifications

• Experience building and maintaining production data pipelines.
• Proficiency in Python for large-scale data processing and pipeline development.
• Strong knowledge of data cloud technologies with hands-on experience on a major cloud platform (GCP, AWS, or Azure).
• Fluency in English with excellent communication skills for technical stakeholders.

Responsibilities

• Design, build, and maintain end-to-end data pipelines processing massive data streams (~100 TB/day).

Skills

Python

English fluency

Data processing

Tools

dbt

Airflow

GCP

BigQuery

Spanner

Pub/Sub

Kubernetes

Claude

Job description

Lighthouse is looking for a Senior Data Engineer to own the end-to-end lifecycle of its large-scale ingestion and transformation engine. You will design, develop, scale, and maintain data pipelines handling massive data streams around 100 TB per day.

You will work with Python, dbt, Airflow on Google Cloud Platform, BigQuery, Spanner, Pub/Sub, Kubernetes, and Claude, collaborating with product, engineering, support, and business teams to deliver high-fidelity data products.