Senior Data Engineer - Lakehouse, AI Pipelines, Remote

AgileEngine; Kraków; On-site; PLN 180,000 - 240,000; Full time

Benefits offered by this job

Remote work with flexible hours

Competitive compensation

Annual learning budget

Mentorship and growth

Job summary

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming data pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. You will design, build, and operate pipelines, mentor junior engineers, and collaborate with cross-functional teams to deliver

Qualifications

• 4+ years of professional data engineering experience with Apache Spark and cloud data architectures.
• Strong hands-on experience building data pipelines with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with data modeling in dimensional and Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with data migrations and data hygiene during porting legacy systems.
• Excellent problem-solving, collaboration, and communication skills including mentoring juniors.

Responsibilities

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics and ML workloads.
• Migrate legacy ETL workloads to Lakehouse with validated data parity and minimal disruption.
• Leverage Claude or Github Copilot to accelerate development, including tests and docs.

Skills

Apache Spark

Python

SQL

Data modeling

Mentoring

Collaboration

Tools

Databricks

Delta Lake

Airflow

Job description

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design batch and streaming data pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role also uses AI tools like Claude and GitHub Copilot to speed up development. You will design, build, and operate pipelines, mentor junior engineers, and collaborate with cross-functional teams to deliver