Senior Data Engineer

N-iX; España; A distancia; EUR 70.000 - 110.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Flexible remote work

Education reimbursement

Career growth

Mentorship program

Team events

Descripción de la vacante

N-iX is seeking a Senior Data Engineer to design, scale, and maintain high-performance ingestion, CDC, and delivery pipelines on Google Cloud Platform. You will architect enterprise-grade data platforms handling metadata, feature extraction, and licensable asset datasets.

You will work across Snowflake, ClickHouse, dbt Core, Airflow, Python, and SQL to deliver low-latency data exposure, delta tracking, and reliable integration endpoints for downstream ML workloads.

Formación

• 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Pub/Sub, GKE, Cloud Storage).
• Deep proficiency with Snowflake and ClickHouse for analytics and integration layers.
• Experience designing CDC pipelines and incremental delta loading at multi-terabyte scale.
• Hands-on with dbt Core and Apache Airflow in an enterprise CI/CD setting.
• Advanced Python and SQL; Go/Java/Scala is a strong plus.

Responsabilidades

• Design and scale ingestion pipelines on GCP and drive high-throughput data flows.
• Architect CDC workflows and incremental processing across platforms.
• Build dbt models and Airflow orchestrations following Data Vault patterns.
• Maintain Snowflake and deploy ClickHouse for low-latency metadata queries.
• Create integration views and APIs to expose real-time delta updates.

Conocimientos

GCP expertise

Python

SQL

Data modeling (Data Vault)

CDC / ETL pipelines

Cloud architecture

Go/Java/Scala (plus)

Herramientas

Snowflake

ClickHouse

dbt Core

Apache Airflow

GitHub Actions

BigQuery

GKE

Pub/Sub

Descripción del empleo

Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets.

This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models.

You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption.

Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL.

Responsibilities

• Ingestion Architecture & GCP Foundation:
• Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
• Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).
• Enterprise ETL, CDC & Incremental Processing:
• Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
• Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
• Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.
• Multi-Engine Analytics & Query Optimization:
• Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
• Deploy and manage ClickHouse as a high-concurrency analytical engine to deliver sub-second metadata retrieval and indexing.
• Construct dedicated integration views and API query layers allowing internal and external clients to consume real-time delta updates.
• Data Quality, Governance & Scalability:
• Enforce metadata schema standardization, canonical asset mapping, and automated validation layers across diverse asset types (images, video, 3D models, audio).
• Collaborate closely with cross-functional teams to integrate metadata pipelines with feature extraction tools, search indexes, and licensing platforms.

Requirements

• Google Cloud Platform (GCP): 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Cloud Pub/Sub, GKE, Google Cloud Storage).
• Database & Query Engines: Deep technical proficiency with both Snowflake (warehousing, semi-structured queries, integration layers) and ClickHouse (real-time aggregation, low-latency columnar storage, indexing).
• Enterprise ETL / CDC: Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte dataset scale.
• Data Transformation & Orchestration: Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (e.g., GitHub Actions).
• Programming Skills: Advanced skills in Python and SQL; familiarity with Go, Java, or Scala is a strong plus.

Nice-to-Have Qualifications

• Experience with Data Vault 2.0 architecture, hash-key generation, and raw/staging/integration layer design.
• Background handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations.
• Familiarity with media metadata schemas (images, 3D model stems, video derivatives).
• Experience with other major cloud providers such as AWS, Azure, or Oracle Cloud.

Key Success Metrics (First 90 Days)

• GCP Pipeline Optimization: Standardize ingestion batching and pipeline compute infrastructure on GCP.
• ClickHouse & Snowflake Integration: Establish streamlined integration views exposing low-latency metadata delta changes.
• Automated CDC Workflows: Finalize automated dbt delta transformations with hash tracking across primary warehouse layers.

We offer*:

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits
• not applicable for freelancers