Data Engineer

Intellias; Portugal; Presencial; EUR 70.000 - 100.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Health Insurance

Annual Leave 22 days

Paid Sick Leave

Permanent Contract

Modern Porto Office

Snacks & Drinks

Discount Platform

Descripción de la vacante

Intellias is hiring a Senior Data Engineer to join a delivery team for a large-scale enterprise data platform migration for a financial client. You will write production code weekly, own architecture for your workstream, and mentor a mid-level engineer.

You will build medallion pipelines on Databricks, register data in Unity Catalog, ensure data quality, and deploy via CI/CD in a Lakehouse environment. This role is based in Porto with a permanent contract and office support.

Formación

• Minimum 6+ years in data engineering with recent 3+ years on Databricks in production.
• Experience with medallion/lakehouse architecture and Delta Lake.
• Strong Python, PySpark and SQL; proficient in Spark UI/Photon performance tuning.
• Experience with streaming ingestion (Kafka) and CDC patterns.
• Experience with data quality at scale and lineage/cataloguing.
• CI/CD for data platforms and IaC (Terraform or equivalent).
• Fluent English for collaboration across teams.

Responsabilidades

• Deliver end-to-end data domain pipelines on Azure Databricks.
• Design Bronze/Silver/Gold pipelines using Lakeflow, Auto Loader, and Delta Lake.
• Register curated data products in Unity Catalog with tags and access policies.
• Implement data quality gates and refine rules with stewards.
• Package and deploy work using Databricks Asset Bundles via CI/CD.
• Write automated tests and documentation; collaborate with architecture team.
• Explore AI tooling integration (Claude Code, Databricks Assistant) for automation.

Conocimientos

Databricks

Lakehouse

Python

PySpark

SQL

Kafka

Data Quality

CI/CD

Terraform

English

Herramientas

Delta Lake

Unity Catalog

Lakeflow

Auto Loader

Descripción del empleo

We are hiring a Senior Data Engineer to join an Intellias delivery team on a large-scale enterprise data platform migration programme for a financial services client. This is a hands-on senior role. You will write production code every week, own architectural decisions on your workstream, and mentor a paired mid-level engineer.

Project Overview:

• Our client is an independent, active global asset manager with over R3 trillion in assets under management. They are executing a firm-wide data strategy to govern, manage and engineer data as a product across more than 70 business-owned sub-domains.
• As part of this, they are migrating to Databricks as the foundational layer of their Enterprise Data Platform, adopting a lakehouse architecture built on open formats, declarative pipelines and Unity Catalog.

Requirements:

• 6+ years of data engineering experience, with at least the last three years on Databricks in production.
• Deep hands-on experience with medallion / Lakehouse architecture, Delta Lake, Unity Catalog, Lakeflow Declarative Pipelines and Databricks Asset Bundles.
• Strong python, PySpark and SQL, comfortable diagnosing performance issues on large workloads through Spark UI and Photon.
• Real production experience with streaming ingestion (Kafka Structured Streaming, Auto Loader) and CDC patterns.
• Experience implementing data quality at scale (Great Expectations or equivalent), plus lineage, cataloguing and access control.
• CI/CD on data platforms through Azure DevOps or GitHub Actions, with infrastructure as code (Terraform or equivalent).
• Fluent working English.

- Nice to have

• Active Databricks certifications
• Prior exposure to investment management platforms, asset management operations data, market data feeds from major providers
• CDMP DAMA certification or equivalent data governance credential.
• Experience applying agentic engineering tooling to production data engineering (not just personal productivity), including MCP-connected Databricks or cloud servers.

Responsibilities:

• Perform the end-to-end delivery of one or more priority data domains on Azure Databricks.
• Design and build medallion (Bronze / Silver / Gold) pipelines using Lakeflow Declarative Pipelines, Auto Loader, Structured Streaming and Delta Lake with Liquid Clustering.
• Register curated data products in Unity Catalog with the correct tags, masks, row filters, lineage and access policies.
• Implement data quality gates at the Silver-to-Gold boundary, and refine rules with domain stewards.
• Package and deploy work using Databricks Asset Bundles through Azure DevOps CI/CD
• Write automated tests and documentation
• Work at AI-as-Collaborator level today with a path to AI-as-Orchestrator, using Claude Code, Databricks Assistant, GitHub Copilot and MCP-connected agent tooling to accelerate the recurring parts of pipeline build, test and deploy.

What we offer:

• 22 days of annual leave per year + 3 paid sick leaves;
• Permanent contract;
• Health Insurance (employee);
• Modern, well-equipped office in the centre of Porto;
• Snacks and drinks at the office;
• Access to discount platform.

Why this position:

Build within a large-scale enterprise data platform migration from day one, designing and delivering the pipelines, ingestion patterns and domain migrations that bring a 70+ sub-domain data strategy to life. You'll work hands-on with the leading edge of the Databricks ecosystem on a high-stakes lakehouse build, applying agentic engineering tooling to accelerate delivery, and working closely with the architecture team to turn platform standards into working, production-grade data.