Senior Data Engineer

Nix; España; Remote; EUR 90,000 - 130,000; Full time

Benefits offered by this job

Flexible working format

Education reimbursement

Career growth opportunities

Professional development tools

Corporate events and team activities

Job summary

Nix is seeking a Senior Data Engineer to design, scale, and maintain high-performance ingestion, CDC, and delivery pipelines on GCP. You will architect enterprise-grade data platforms processing large-scale metadata and feature extraction datasets for AI/ML use, leveraging Snowflake, ClickHouse, dbt Core, and Airflow.

The role emphasizes building robust Delta tracking, real-time exposure APIs, and governance across diverse asset types, with a focus on scalable, multi-cloud data processing

Qualifications

• 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Pub/Sub, GKE, Google Cloud Storage).
• Deep technical proficiency with Snowflake and ClickHouse for real-time and analytical workloads.
• Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte scales.
• Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (GitHub Actions).
• Advanced Python and SQL; familiarity with Go, Java, or Scala is a strong plus.

Responsibilities

• Ingestion Architecture & GCP Foundation: design and optimize scalable ingestion pipelines on GCP.
• ETL, CDC & Incremental Processing: build CDC and delta workflows with dbt and Airflow.
• Multi-Engine Analytics & Query Optimization: maintain Snowflake integrations and deploy ClickHouse.
• Data Quality, Governance & Scalability: enforce metadata standards and integrate with feature extraction tools.

Skills

Python

SQL

Data pipelines

Data modeling

Go

Java

Scala

Tools

GCP

Snowflake

ClickHouse

dbt Core

Apache Airflow

GitHub Actions

BigQuery

Dataflow

Pub/Sub

GKE

Job description

Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets.

This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models.

You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption.

Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL.

Responsibilities

• Ingestion Architecture & GCP Foundation:
  • Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
  • Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).
• Enterprise ETL, CDC & Incremental Processing:
  • Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
  • Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
  • Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.
• Multi-Engine Analytics & Query Optimization:
  • Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
  • Deploy and manage ClickHouse as a high-concurrency analytical engine to deliver sub-second metadata retrieval and indexing.
  • Construct dedicated integration views and API query layers allowing internal and external clients to consume real-time delta updates.
• Data Quality, Governance & Scalability:
  • Enforce metadata schema standardization, canonical asset mapping, and automated validation layers across diverse asset types (images, video, 3D models, audio).
  • Collaborate closely with cross-functional teams to integrate metadata pipelines with feature extraction tools, search indexes, and licensing platforms.

Requirements

• Google Cloud Platform (GCP): 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Cloud Pub/Sub, GKE, Google Cloud Storage).
• Database & Query Engines: Deep technical proficiency with both Snowflake (warehousing, semi-structured queries, integration layers) and ClickHouse (real-time aggregation, low-latency columnar storage, indexing).
• Enterprise ETL / CDC: Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte dataset scale.
• Data Transformation & Orchestration: Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (e.g., GitHub Actions).
• Programming Skills: Advanced skills in Python and SQL; familiarity with Go, Java, or Scala is a strong plus.

Nice-to-Have Qualifications

• Experience with Data Vault 2.0 architecture, hash-key generation, and raw/staging/integration layer design.
• Background handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations.
• Familiarity with media metadata schemas (images, 3D model stems, video derivatives).
• Experience with other major cloud providers such as AWS, Azure, or Oracle Cloud.

Key Success Metrics (First 90 Days)

• GCP Pipeline Optimization: Standardize ingestion batching and pipeline compute infrastructure on GCP.
• ClickHouse & Snowflake Integration: Establish streamlined integration views exposing low-latency metadata delta changes.
• Automated CDC Workflows: Finalize automated dbt delta transformations with hash tracking across primary warehouse layers.

We offer*

• Flexible working format - remote, office-based or flexible
• A competitive salary and good compensation package
• Personalized career growth
• Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
• Active tech communities with regular knowledge sharing
• Education reimbursement
• Memorable anniversary presents
• Corporate events and team buildings
• Other location-specific benefits

*not applicable for freelancers