Senior Data Engineer - Databricks Lakehouse & AI Pipelines

AgileEngine; Poznań; On-site; PLN 180,000 - 300,000; Full time

Benefits offered by this job

Growth opportunities

Competitive compensation

Remote work

Modern projects

Collaborative culture

Well-being programs

Job summary

AgileEngine is seeking an experienced Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting, and ML.

You will leverage AI tools like Claude and GitHub Copilot to speed up development, write tests, and create documentation. Strong collaboration across teams and a focus on scalable, secure data platforms are essential.

Qualifications

• 4+ years of professional data engineering experience with Apache Spark and cloud-based architectures.
• Strong hands-on data pipelines with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with dimensional and Lakehouse data modeling.
• Experience with streaming ingestion (Structured Streaming, Kafka, Event Hubs).
• Experience with data governance and data quality controls.

Responsibilities

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics and ML.
• Migrate legacy ETL workloads to the Databricks Lakehouse with validated data parity.
• Use Claude or Copilot to accelerate development, write tests, and create docs.
• Write clean Python and SQL with high-quality code reviews and documentation.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance using Unity Catalog and automated checks.
• Debug and resolve pipeline failures and production incidents.
• Collaborate in Agile delivery with DevOps and analytics teams.

Skills

Apache Spark

PySpark

Delta Lake

Databricks

SQL

Python

Data modeling

Governance

CI/CD

Tools

Databricks

Airflow

Azure Data Factory

Terraform

Job description

AgileEngine is seeking an experienced Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture for analytics, reporting, and ML.

You will leverage AI tools like Claude and GitHub Copilot to speed up development, write tests, and create documentation. Strong collaboration across teams and a focus on scalable, secure data platforms are essential.