Senior Data Engineer, Streaming & Lakehouse Platform

Deel; México; Vor Ort; MXN 2.242.000 - 3.363.000; Vollzeit

Benefits dieser Stelle

Stock grants

Flexible office membership

Disability accommodations

Zusammenfassung

Deel is building a world-class Data Platform team that turns operational data into rapid, data-driven insights. You will design streaming jobs, run compute clusters, and keep lakehouse tables healthy, making data models accessible to engineering teams across the company.

You’ll tackle end-to-end streaming challenges, from CDC semantics to scalable deployment, while improving observability and developer experience. This is a chance to shape data-driven decisions at a global SaaS company.

Qualifikationen

• 7+ years in data engineering, including 3+ years streaming pipelines in production.
• Hands-on Spark Structured Streaming, state stores, checkpointing, and tuning.
• Experience with lakehouse formats (Iceberg preferred) and Snowflake/Delta/Hudi.
• Operate Spark on AWS EMR/EKS, with autoscaling and capacity planning.
• Solid CDC and Kafka semantics, including offsets and delivery guarantees.
• Strong Python and SQL; Scala/Java is a plus.
• Experience with Airflow, CI/CD and infra-as-code.
• Methodical debugging: read logs, form hypotheses, prove fixes with data.

Aufgaben

• Design, build, and harden Spark Structured Streaming apps processing CDC events to near real-time data models.
• Run and scale AWS EMR clusters; manage sizing, autoscaling, multi-cluster deployment, memory tuning.
• Own health of Iceberg tables; partition specs, compaction, expiry, cleanup at scale.
• Onboard new domains to streaming platform; configure CDC sources/destinations, verify backfills.
• Debug production issues across stack; fix root causes, not symptoms.
• Build observability and tooling to shorten diagnosis time; logs to Grafana Loki, runbooks.
• Improve deployment/DevEx via CI/CD, config structure, and guardrails for safe self-serve.
• Handle ad hoc ingestion/backfill requests and escalations from partner teams.

Kenntnisse

Spark Structured Streaming

Python

SQL

Kafka semantics

CDC handling

CI/CD

Debugging

AWS EMR

Tools

Apache Iceberg

Snowflake

Estuary Flow

Airflow

Grafana Loki

Kubernetes

CI/CD tooling

Jobbeschreibung

Deel is building a world-class Data Platform team that turns operational data into rapid, data-driven insights. You will design streaming jobs, run compute clusters, and keep lakehouse tables healthy, making data models accessible to engineering teams across the company.

You’ll tackle end-to-end streaming challenges, from CDC semantics to scalable deployment, while improving observability and developer experience. This is a chance to shape data-driven decisions at a global SaaS company.