Data Architect: CDC & Databricks (Lisbon Hybrid)

AdvanceWorks; Lisboa; Hybrid; EUR 60,000 - 100,000; Full time

Job summary

AdvanceWorks in Lisbon is seeking a Data Architect to join our team and help evolve ingestion patterns, CDC, and a scalable data architecture. You will design and implement Databricks-based data pipelines across Azure, Spark, Delta Lake, and Kafka for a global organization.

You will collaborate with Data Engineers and cross-functional teams to ensure data integrity, traceability, and performance, while aligning with best practices in ETL/ELT and cloud-native architectures.

Qualifications

• 3+ years of experience in Data Engineering or related data-intensive roles.
• Solid hands-on experience with Databricks, including notebooks, clusters, jobs, and workflows.
• Strong knowledge of Apache Spark, including batch processing and distributed data transformations.
• Strong experience with Python and SQL.
• Hands-on experience with Delta Lake and modern lakehouse architectures.
• Practical understanding of Change Data Capture (CDC) concepts and implementation patterns.
• Experience developing and parametrizing data pipelines and processing frameworks.
• Experience working with cloud data platforms, particularly Microsoft Azure.
• Familiarity with Azure services such as Azure File Share and Data Lake.
• Experience integrating data pipelines with Kafka, ideally Kafka Confluent.
• Good understanding of data architecture and ETL/ELT best practices.
• Experience designing scalable and resilient data processing solutions.
• Strong problem-solving skills and ability to troubleshoot distributed data processing environments.
• Strong communication skills and ability to collaborate with different technical teams.
• Availability to work in a hybrid model, 3 days per week in Tagus Park.
• Fluent Portuguese and English, both written and spoken.

Responsibilities

• Developing and maintaining Change Data Capture (CDC) applications and data ingestion processes in Databricks.
• Creating and maintaining Databricks notebooks for setup, configuration, and data processing.
• Developing and parametrizing batch processing pipelines using Apache Spark.
• Implementing CDC logic to correctly detect and process inserts, updates, and deletes.
• Ensuring the generation of complete data snapshots while maintaining historical changes and data traceability.
• Working with Delta Lake to support reliable, scalable, and performant data processing.
• Working with multiple data sources, including Azure File Share, DB2, SQL Server, and SingleStore.
• Integrating processed data with destinations such as SingleStore and Kafka Confluent.
• Working with Databricks Workflows and Azure services to orchestrate and operationalize data pipelines.
• Optimizing processing performance and ensuring that pipelines are scalable and resilient.
• Contributing to the definition and evolution of data ingestion architecture and processing frameworks.
• Collaborating with other Data Engineers and technical teams to integrate processed data with downstream microservices.
• Ensuring data integrity, consistency, and traceability throughout migration and processing flows.
• Applying best practices around ETL/ELT, data engineering, monitoring, error handling, and pipeline reliability.
• Contributing to continuous improvement initiatives across the data platform and its ingestion frameworks.

Skills

Databricks

Apache Spark

Python

SQL

Delta Lake

CDC concepts

Azure

Kafka

Data ingestion

Tools

Databricks notebooks

Azure services

Kafka Confluent

Job description

AdvanceWorks in Lisbon is seeking a Data Architect to join our team and help evolve ingestion patterns, CDC, and a scalable data architecture. You will design and implement Databricks-based data pipelines across Azure, Spark, Delta Lake, and Kafka for a global organization.

You will collaborate with Data Engineers and cross-functional teams to ensure data integrity, traceability, and performance, while aligning with best practices in ETL/ELT and cloud-native architectures.