Senior Data Engineer — Real-Time Streaming & Lakehouse

Deel; México; On-site; MXN 2,171,000 - 3,257,000; Full time

Job summary

Deel is transforming how global data is turned into actionable insights. Our Data Platform team builds streaming pipelines using Spark Structured Streaming and Apache Iceberg to deliver near-real-time models.

You’ll own the platform end-to-end, from design to deployment, driving cost efficiency and scalability across domains. You’ll collaborate with partners to onboard new data sources, optimize lakehouse performance, and enhance observability with modern tooling.

Qualifications

• 7+ years of data engineering experience, with at least 3 years running streaming pipelines in production.
• Deep hands-on experience with Apache Spark, especially Structured Streaming, state stores, and performance tuning.
• Production experience with a lakehouse table format, ideally Apache Iceberg; familiarity with partitioning strategies.

Responsibilities

• Design, build, and harden Spark Structured Streaming applications that process CDC events into near-real-time data models.
• Run and scale AWS EMR clusters, manage sizing, multi-cluster deployment, and cost optimization.
• Own health of Apache Iceberg tables with partition specs, compaction, and cleanup at scale.
• Onboard new business domains to the streaming platform and coordinate with source-database owners.
• Debug production issues, identify root causes, and implement durable fixes with data-backed validation.
• Build observability tooling and runbooks to speed up diagnosis and recovery.
• Improve deployment and developer experience via CI/CD pipelines and guardrails.
• Handle ad hoc ingestion and backfill requests from partner teams

Skills

Spark Structured Streaming

Python

SQL

Scala/Java

CDC/Kafka semantics

Cloud/AWS

Tools

Apache Iceberg

Snowflake

Estuary Flow

Kafka

Airflow

Grafana Loki

Kubernetes (K8s)

Job description

Deel is transforming how global data is turned into actionable insights. Our Data Platform team builds streaming pipelines using Spark Structured Streaming and Apache Iceberg to deliver near-real-time models.

You’ll own the platform end-to-end, from design to deployment, driving cost efficiency and scalability across domains. You’ll collaborate with partners to onboard new data sources, optimize lakehouse performance, and enhance observability with modern tooling.