Senior Data Engineer - Databricks Lakehouse (Remote)

AgileEngine; Porto; Presencial; EUR 65 000 - 90 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Growth without limits

Competitive compensation

Flexibility: fully remote

Resumo da oferta

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design and run batch and streaming pipelines with PySpark and Delta Lake across bronze, silver, and gold layers, while leveraging AI tools to speed development.

You will migrate legacy workloads, implement data governance with Unity Catalog, and collaborate with DevOps and analytics teams to ensure reliability, security, and scalability across global data

Qualificações

• :4+ years of professional data engineering with Spark and cloud data architectures.
• Hands-on experience building data pipelines with Databricks, PySpark, and Delta Lake.
• Advanced SQL and Python with strong data modeling skills across dimensional and Lakehouse patterns.
• Experience with streaming ingestion using Structured Streaming, Auto Loader, Kafka, or Event Hubs.
• Experience with workflow orchestration (Databricks Workflows, Airflow, or Azure Data Factory).
• Experience with legacy platform migrations, ETL modernization, or data hygiene when porting old systems.
• Strong problem-solving, collaboration, and communication skills, including mentoring junior engineers and explaining data concepts to non-technical stakeholders.
• Familiarity with Unity Catalog, data governance, access control, and PII handling.
• Experience with dbt or an equivalent transformation framework.

Responsabilidades

• Design, build, and operate batch and streaming data pipelines on Databricks using PySpark, Delta Lake, and Databricks Workflows.
• Model and maintain a medallion (bronze/silver/gold) architecture serving analytics, reporting, and machine learning consumers.
• Migrate legacy ETL and data warehouse workloads onto the Lakehouse with validated data parity and minimal business disruption.
• Use Claude or Github Copilot as a development accelerator, generating code scaffolding, writing and reviewing tests, creating documentation and prototyping solutions.
• Write clean, well-tested Python and SQL; maintain high standards through code review and documentation.
• Optimize Spark jobs and Delta tables for performance and cost, including partitioning, clustering, caching, and cluster sizing.
• Implement data quality, lineage, and governance controls using Unity Catalog and automated validation checks.
• Debug, troubleshoot, and resolve pipeline failures, data defects, and production incidents.
• Collaborate with DevOps, platform, and analytics engineers on observability, security, and compliance best practices.

Conhecimentos

English (Upper-Intermediate)

Problem solving

Communication

Ferramentas

Apache Spark

PySpark

Databricks

Delta Lake

Structured Streaming

Kafka

Airflow

Azure Data Factory

SQL

Python

Databricks Workflows

Descrição da oferta de emprego

AgileEngine is seeking a Senior Data Engineer to modernize a 15-year-old data warehouse into a governed Databricks Lakehouse. You will design and run batch and streaming pipelines with PySpark and Delta Lake across bronze, silver, and gold layers, while leveraging AI tools to speed development.

You will migrate legacy workloads, implement data governance with Unity Catalog, and collaborate with DevOps and analytics teams to ensure reliability, security, and scalability across global data