Data Engineer – Lakehouse Core & Governance

Pentasia; México; Remoto; MXN 700.000 - 1.000.000; Tempo pieno

Descrizione del lavoro

Pentasia is seeking a data engineer to own a data lake and lakehouse architecture, supporting millions of financial events daily across users. You’ll decide how data lands, is stored, retained, and governed on S3 and Snowflake/Databricks to ensure accurate, reconciled data with zero drift from source.

You’ll implement bronze/silver/gold layers, enable CDC streaming, optimize partitioning and query performance, and oversee governance, retention, and cost control.

Competenze

• 3+ years in data engineering, with real ownership of a large-scale data lake or lakehouse.
• Experience with cloud object storage (S3/GCS) plus Snowflake, Databricks, or BigQuery in production.
• Hands-on with an open table format (Iceberg, Delta, or Hudi).
• Deep understanding of partitioning, file layout, and query optimization at terabyte-plus scale.
• CDC and streaming ingestion experience (Kafka + Debezium or equivalent).
• Strong SQL and data modeling; grounding to reason about OLTP systems you capture from.
• Data lake governance: catalogs, lineage, access control, PII, retention.
• Python and an orchestrator (Airflow/Dagster) as tools — architecture and ownership role.

Mansioni

• Own the lakehouse architecture: bronze/silver/gold layers, Iceberg/Delta tables, schema evolution.
• Land operational data via CDC streaming (Kafka, Debezium), handle late and duplicate events.
• Design data layout for speed and cost: partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake.
• Own retention and archival: storage tiering, regulatory retention, immutability, GDPR deletion.
• Guarantee correctness: freshness SLAs, drift detection, reconciliation against source wallet and ledger systems.
• Own governance: catalog and lineage, row/column access control, PII masking, encryption, audit trails.
• Monitor ingestion health, data anomalies, and cloud storage/compute spend.

Conoscenze

Data lake

Lakehouse

S3

Snowflake

Databricks

Iceberg/Delta/Hudi

Kafka

Debezium

SQL

Data modeling

Partitioning

Python

Airflow

Dagster

Strumenti

Airflow

Dagster

Descrizione del lavoro

Pentasia is seeking a data engineer to own a data lake and lakehouse architecture, supporting millions of financial events daily across users. You’ll decide how data lands, is stored, retained, and governed on S3 and Snowflake/Databricks to ensure accurate, reconciled data with zero drift from source.

You’ll implement bronze/silver/gold layers, enable CDC streaming, optimize partitioning and query performance, and oversee governance, retention, and cost control.