Data platform engineer for regulated environments

Stefanini EMEA; Lisboa; On-site; EUR 70,000 - 110,000; Full time

Benefits offered by this job

Private health insurance

Career development opportunities

Referral bonuses

Job summary

Stefanini EMEA ищет опытного Senior Data Platform Engineer для разработки модульной платформы данных на базе Databricks. Вы будете заниматься ingestion, governance, data modeling и созданием data products на базе единой canonical модели.

Ваша работа охватит конвейеры данных, обеспечение качества и безопасности данных, интеграцию сервисов API/GraphQL и развитие платформы с акцентом на Data as a Product и self-service для команд по данным.

Qualifications

• 6+ лет в data engineering
• Опыт проектирования/эксплуатации Databricks Lakehouse на продуктиве
• Практический опыт с Spark, Delta Lake и Delta Live Tables/Lakeflow — большой плюс
• Опыт реализации data governance, API/GraphQL exposure
• Знания GDPR/CCPA, Data as a Product подход

Responsibilities

• Разработка и поддержка модульных ingestion patterns (batch, API, event-driven)
• Построение Databricks-пайплайнов по архитектуре Bronze/Silver/Gold
• Внедрение контроля доступа, lineage, каталогов и аудита
• Упор на data quality, data contracts и мониторинг
• Защита данных: маскирование и псевдонимизация в регуляторных условиях
• Сопровождение data products и обеспечение SLA
• Сотрудничество с командами доменов и обеспечения соответствия

Skills

Python

Scala

SQL

Tools

Databricks

Delta Lake

Unity Catalog

Terraform

Job description

Описание:

This client is building a modular, enterprise-grade data platform powered by Databricks, designed to deliver trusted, governed data through multiple consumption channels, including analytics, APIs, streaming, and knowledge graph solutions. The platform is built on a shared canonical data model and emphasizes scalability, governance, privacy, and a Data as a Product mindset. As a Senior Data Platform Engineer, you will play a key role in designing and evolving the platform across ingestion, governance, data modeling, and data products.

Задачи:

Develop and maintain modular ingestion patterns supporting batch, API-based, and event-driven data sources (e.g., JDBC, SaaS/ERP connectors, Kafka, CDC). Establish reusable ingestion frameworks (metadata driven when possible) that enable rapid and consistent onboarding of new data sources. Build and optimize Databricks-based data pipelines (Spark, Delta Live Tables/Lakeflow) across the medallion architecture (Bronze, Silver, Gold). Implement and evolve data governance capabilities, including access controls, lineage, cataloging, auditing, data classification, and privacy compliance. Define and enforce data quality standards, data contracts, monitoring, and alerting across ingestion and transformation layers. Implement data protection and privacy solutions, including masking, tokenization, pseudonymization, and other appropriate anonymization techniques to meet regulatory requirements (e.g. GDPR). Build, own, and evolve data products aligned to a canonical data model, ensuring clear ownership, SLAs and documentation. Collaborate with domain teams to integrate source data into the canonical model while preserving domain-specific nuance. Champion a Data as a Product approach: discoverability, self-service, versioning, and consumer-facing contracts. Build and maintain data consumption and serving layers, including BI/analytics datasets, APIs (REST/GraphQL), and use case-driven extensions such as streaming outputs or knowledge graph representations. Drive technical standards and platform direction across CI/CD, Infrastructure as Code, testing, observability, and cost optimization Mentor engineers and partner with governance, security, architecture, and business stakeholders.

Требования:

Around 6+ years in data engineering, with 2+ years designing/operating Databricks-based Lakehouse platforms at production scale. Hands-on expertise with Apache Spark and Delta Lake; experience with Delta Live Tables/Lakeflow Declarative Pipelines is a strong plus. Practical experience with Unity Catalog (or equivalent) for access control, lineage, and cataloguing. Demonstrated experience implementing data anonymisation/pseudonymisation techniques in a regulated environment. Solid understanding of canonical/conformed data modelling (e.g., data vault, dimensional modelling, domain-driven design for data). Track record exposing data via APIs (REST/GraphQL) - designing schemas, contracts, and versioning for external/internal consumers. Proficient in Python and/or Scala, SQL, and infrastructure-as-code (Terraform preferred). Strong grasp of data governance frameworks, data privacy regulation (GDPR/CCPA), and data security practices. Experience with cloud platforms (Azure, AWS, or GCP) and their native data/event services. Будет плюсом: Exposure to Structured Streaming or event-driven ingestion (Kafka, CDC tools such as Debezium) - not a core requirement, but useful as the platform matures; Familiarity with knowledge graph concepts (e.g., RDF/property graphs, Neo4j, graph layers on Spark) - a plus rather than an expectation; Experience with data mesh or federated data product architectures; Experience with metadata management/catalog tools beyond Unity Catalog (e.g., OpenMetadata, Collibra, Purview); Exposure to MLOps or feature store patterns on Databricks; Relevant certifications (Databricks Data Engineer Professional, cloud architecture certs)

Условия:

You will find here not only a challenging and interesting workplace, but also a rewarding work experience, with competitive compensation and benefit packages: Private health insurance; Career development: opportunity to grow within the team; Special referral bonuses for recommending your friends; Friendly team who is eager to meet you;