Senior Data Engineer — Databricks Lakehouse (Remote)

AgileEngine; Wrocław; On-site; PLN 180,000 - 320,000; Full time

Benefits offered by this job

Growth without limits

Competitive compensation

Flexibility: 100% remote with flexible

Modern projects with global teams

Collaborative culture

Well-being & support

Job summary

AgileEngine is seeking a Senior Data Engineer to modernize a legacy data warehouse into a Databricks Lakehouse. You will design and operate batch and streaming data pipelines, applying a medallion architecture to serve analytics and ML workloads.

You will leverage PySpark, Delta Lake, and Databricks Workflows while gradually migrating legacy workloads with data parity. Collaboration across teams and adherence to governance are essential.

Qualifications

• 4+ years of professional data engineering experience with Apache Spark.
• Hands-on with Databricks, PySpark, and Delta Lake.
• Advanced SQL and Python with dimensional and Lakehouse modeling.
• Experience streaming ingestion via Structured Streaming, Kafka, or Event Hubs.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience migrating legacy ETL to Lakehouse.
• Strong collaboration and mentoring skills.
• Upper-intermediate English.

Responsibilities

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and ML consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
• Use Claude or Github Copilot as a development accelerator, generating code scaffolding, writing and reviewing tests, creating documentation and prototyping solutions.
• Write clean, well-tested Python and SQL; maintain high standards through code review and documentation.
• Optimize Spark jobs and Delta tables for performance and cost, including partitioning, clustering, caching, and cluster sizing.
• Implement data quality, lineage, and governance controls using Unity Catalog and automated validation checks.
• Debug, troubleshoot, and resolve pipeline failures, data defects, and production incidents.
• Participate in Agile or product-centric delivery practices including sprint planning and retrospectives.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices.

Skills

Apache Spark

Databricks

Delta Lake

Python

SQL

Tools

Databricks Workflows

Airflow

Azure Data Factory

Terraform

GitHub Copilot

Job description

AgileEngine is seeking a Senior Data Engineer to modernize a legacy data warehouse into a Databricks Lakehouse. You will design and operate batch and streaming data pipelines, applying a medallion architecture to serve analytics and ML workloads.

You will leverage PySpark, Delta Lake, and Databricks Workflows while gradually migrating legacy workloads with data parity. Collaboration across teams and adherence to governance are essential.