Remote Senior Data Engineer: Databricks Lakehouse Pipelines

AgileEngine; Lisboa; Presencial; EUR 103 000 - 155 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Growth opportunities

Competitive compensation

Remote-friendly

Resumo da oferta

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design and implement batch and streaming pipelines using PySpark, Delta Lake, and Databricks Workflows.

You will migrate legacy ETL workloads, ensure data parity, and collaborate with DevOps and analytics teams to uphold governance, quality, and performance at scale.

Qualificações

• 4+ years in data engineering with Spark and cloud data architectures.
• Hands-on with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python with data modeling across dimensional and Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience migrating legacy platforms and ETL modernization.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion architecture (bronze/silver/gold) for analytics and ML workloads.
• Migrate legacy ETL workloads to the Lakehouse with data parity and minimal disruption.
• Leverage Claude or Github Copilot to accelerate development, including tests and documentation.
• Write clean Python and SQL, maintain quality via reviews and tests.
• Optimize Spark jobs and Delta tables for performance and cost (partitioning, clustering, caching).
• Implement data quality, lineage, and governance using Unity Catalog and automated checks.
• Debug and resolve production data issues and outages.
• Collaborate in Agile or product-centric delivery with DevOps and analytics teams.

Conhecimentos

Databricks

PySpark

Delta Lake

SQL

Python

Data modeling

English comms

Ferramentas

Databricks

Airflow

Terraform

CI/CD

Unity Catalog

Descrição da oferta de emprego

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design and implement batch and streaming pipelines using PySpark, Delta Lake, and Databricks Workflows.

You will migrate legacy ETL workloads, ensure data parity, and collaborate with DevOps and analytics teams to uphold governance, quality, and performance at scale.