Mid Data Engineer

Commit; Warszawa; Hybrid; PLN 180,000 - 270,000; Full time

Job summary

Commit is seeking a Middle Data Engineer to own our data lake and lakehouse architecture, spanning bronze/silver/gold layers and Iceberg/Delta tables. You’ll land data via CDC streaming and ensure accurate, governance-driven data across AWS S3/Snowflake/Databricks for analytics, finance and regulatory teams.

Hybrid work in Kraków with 2 days onsite. You will optimize storage and processing for TB+ scale, enforce retention and GDPR rules, and maintain data quality with drift detection and

Qualifications

• 3+ years hands-on in a production lakehouse environment.
• Lakehouse architecture with bronze/silver/gold, open table formats, schema evolution.
• Data layout & query optimization at TB+ scale with partitioning, compaction, and sizing.
• Cloud lakehouse/DWH in production using Snowflake, Databricks, or BigQuery.
• CDC & streaming ingestion with Kafka + Debezium, handling late/duplicate events.
• Strong SQL and data modeling for financial ledgers.
• Governance: catalogs, lineage, access control, PII masking, retention.
• Cloud object storage (S3 or GCS) with tiering/archival.
• Python and an orchestrator (Airflow or Dagster).

Responsibilities

• Own the lakehouse architecture and data models for reliability and cost.
• Land operational data via CDC streaming, handle out-of-order events.
• Design data layout for speed and cost; optimize queries on major engines.
• Own retention, archival, and regulatory compliance incl. GDPR deletion.
• Ensure data freshness, drift detection, and reconciliation with source systems.
• Governance of catalogs, lineage, access controls, and audit trails.
• Monitor ingestion health, anomalies, and cloud spend.

Skills

Lakehouse

SQL mastery

CDC streaming

Python

Airflow

Tools

Snowflake

Databricks

Trino/Athena/Snowflake

Job description

We’re looking for a Middle Data Engineer to take ownership of our data lake — the system of record for millions of financial events every day, including bets, wallet transactions, and live odds, serving 12M+ active users. You’ll be responsible for designing and maintaining reliable data pipelines and defining how data is ingested, stored, retained, reconciled, and governed across AWS S3 and Snowflake/Databricks. Your work will ensure that Analytics, Finance, and Regulatory teams have access to accurate, consistent, and fully traceable data — with zero drift from source systems.

Location: Kraków, Poland. Hybrid — 2 days per week from the office.

What you will do:

• Own the lakehouse architecture: bronze/silver/gold layers, Iceberg/Delta tables, schema evolution.
• Land operational data via CDC streaming (Kafka, Debezium), handling late and duplicate events.
• Design data layout for speed and cost: partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake.
• Own retention and archival: storage tiering, regulatory retention, immutability, GDPR deletion.
• Guarantee correctness: freshness SLAs, drift detection, reconciliation against the source wallet and ledger systems.
• Own governance: catalog and lineage, row/column access control, PII masking, encryption, audit trails.
• Monitor ingestion health, data anomalies, and cloud storage/compute spend.

Requirements:

Must-have:

• 3+ years hands-on in a production lakehouse environment.
• Lakehouse architecture — bronze/silver/gold layering, an open table format (Iceberg, Delta, or Hudi), schema evolution.
• Data layout & query optimization at TB+ scale — partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake.
• Cloud lakehouse/DWH in production — Snowflake, Databricks, or BigQuery.
• CDC & streaming ingestion — Kafka + Debezium or equivalent; late, duplicate and out-of-order events.
• Strong SQL and data modeling — enough relational grounding to reason about the OLTP systems you capture from. Critical for financial ledgers.
• Correctness — freshness SLAs, drift detection, reconciliation against source wallet/ledger systems.
• Governance — catalogs, lineage, row/column access control, PII masking, retention, GDPR deletion.
• Cloud object storage — S3 or GCS, plus storage tiering and archival.
• Python and an orchestrator — Airflow or Dagster, as tools.

Nice to have:

• Fintech, iGaming, or another regulated, audit-heavy environment.
• Cost monitoring / FinOps for storage and compute spend.
• Hudi specifically; Dagster specifically.
• Immutability / WORM regulatory retention.