Senior Data Engineer: Real-Time Spark & Lakehouse

Deel; Chile; Híbrido; CLP 144.370.000 - 202.117.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Stock options

Flexible working office membership

Learning & growth stipends

Descripción de la vacante

Deel is building a Data Platform to turn operational data into fast, reusable data products. You’ll design streaming jobs, manage compute clusters, and maintain healthy lakehouse tables, enabling near‑real‑time analytics for global payroll and HR workflows.

You’ll own end‑to‑end streaming pipelines, collaborate with data teams, optimize costs, and ship scalable data models that power Deel’s analytics and product decisions. This role offers impactful work in a fast‑growing SaaS environment.

Formación

• 7+ years of data engineering experience, with at least 3 years running streaming pipelines in production.
• Deep hands-on experience with Apache Spark, especially Structured Streaming, state stores, checkpointing, and performance tuning.
• Production experience with a lakehouse table format like Iceberg, including partitioning and maintenance.
• Strong Python and SQL skills; Scala or Java a plus.
• Experience with CDC and Kafka semantics, including offsets and delivery guarantees.
• Experience with Airflow or similar orchestrator, plus CI/CD and IaC practices.
• Methodical debugging style: read logs, form hypotheses, prove fixes with data.

Responsabilidades

• Design, build, and harden Spark Structured Streaming apps processing CDC events into near-real-time data models.
• Run and scale AWS EMR clusters, manage sizing, scaling, and memory tuning to cut costs.
• Own the health of Iceberg tables: partition specs, compaction, snapshot expiry, and cleanups at scale.
• Onboard new business domains to the streaming platform; configure CDC sources/destinations and verify backfills.
• Debug production issues end-to-end, addressing root causes like offset regressions and schema mismatches.
• Build observability and tooling, shipping logs to Grafana Loki, and developing runbooks.
• Improve deployment and developer experience with CI/CD pipelines and infrastructure-as-code.
• Handle ad hoc ingestion/backfill requests from partner teams.

Conocimientos

Spark Structured Streaming

Python

SQL

CDC & Kafka

AWS EMR

Iceberg

CI/CD / IaC

Snowflake

Herramientas

Apache Iceberg

Kafka

Snowflake

Airflow

CI/CD pipelines

Kubernetes

Descripción del empleo

Deel is building a Data Platform to turn operational data into fast, reusable data products. You’ll design streaming jobs, manage compute clusters, and maintain healthy lakehouse tables, enabling near‑real‑time analytics for global payroll and HR workflows.

You’ll own end‑to‑end streaming pipelines, collaborate with data teams, optimize costs, and ship scalable data models that power Deel’s analytics and product decisions. This role offers impactful work in a fast‑growing SaaS environment.