Senior Data Engineer — Remote Lakehouse & AI Pipelines

AgileEngine; Aveiro; Presencial; EUR 80 000 - 120 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Growth opportunities

Competitive compensation

Remote-friendly with flexible hours

Modern projects with global brands

Collaborative culture

Well-being programs

Resumo da oferta

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role involves using AI tools like Claude and GitHub Copilot to speed up development, writing Python and SQL with strong governance and tests, and collaborating across DevOps, platform, and analytics teams.

Qualificações

• 4+ years of professional data engineering with Spark and cloud data architectures.
• Hands-on with Databricks, Spark (PySpark), and Delta Lake.
• Advanced SQL and Python; strong data modeling for Lakehouse patterns.
• Experience with Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with Databricks Workflows, Airflow, or Azure Data Factory.
• Experience migrating legacy ETL to Lakehouse with data hygiene.
• Strong problem solving and ability to mentor juniors.
• Familiarity with Unity Catalog, data governance, and PII handling.
• Experience with dbt or equivalent transformation framework.
• Upper-intermediate English

Responsabilidades

• Design, build, and operate batch and streaming pipelines on Databricks using PySpark and Delta Lake.
• Model and maintain a medallion architecture serving analytics and ML consumers.
• Migrate legacy ETL workloads to Lakehouse with minimal business disruption.
• Use Claude or Copilot to accelerate development, tests, and docs.
• Write clean Python and SQL; ensure code quality through reviews.
• Optimize Spark jobs and Delta tables for performance and cost.
• Implement data quality, lineage, and governance with Unity Catalog.
• Debug, troubleshoot, and resolve pipeline issues in production.
• Collaborate with DevOps, platform, and analytics engineers on observability and security.
• Participate in Agile delivery practices and sprint ceremonies.

Conhecimentos

Apache Spark

PySpark

Delta Lake

Databricks

SQL

Python

Airflow

Unity Catalog

Kafka

Test automation

Ferramentas

Databricks Workflows

Azure Data Factory

Terraform

GitHub Copilot

Descrição da oferta de emprego

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will build batch and streaming pipelines with PySpark and Delta Lake, following a medallion architecture across bronze, silver, and gold layers.

This role involves using AI tools like Claude and GitHub Copilot to speed up development, writing Python and SQL with strong governance and tests, and collaborating across DevOps, platform, and analytics teams.