Senior Data Engineer

AdvanceWorks; Lisboa; Híbrido; EUR 60 000 - 100 000; Tempo integral

Resumo da oferta

AdvanceWorks in Lisbon is seeking a Data Architect to join our team and help evolve ingestion patterns, CDC, and a scalable data architecture. You will design and implement Databricks-based data pipelines across Azure, Spark, Delta Lake, and Kafka for a global organization.

You will collaborate with Data Engineers and cross-functional teams to ensure data integrity, traceability, and performance, while aligning with best practices in ETL/ELT and cloud-native architectures.

Qualificações

• 3+ years of experience in Data Engineering or related data-intensive roles.
• Solid hands-on experience with Databricks, including notebooks, clusters, jobs, and workflows.
• Strong knowledge of Apache Spark, including batch processing and distributed data transformations.
• Strong experience with Python and SQL.
• Hands-on experience with Delta Lake and modern lakehouse architectures.
• Practical understanding of Change Data Capture (CDC) concepts and implementation patterns.
• Experience developing and parametrizing data pipelines and processing frameworks.
• Experience working with cloud data platforms, particularly Microsoft Azure.
• Familiarity with Azure services such as Azure File Share and Data Lake.
• Experience integrating data pipelines with Kafka, ideally Kafka Confluent.
• Good understanding of data architecture and ETL/ELT best practices.
• Experience designing scalable and resilient data processing solutions.
• Strong problem-solving skills and ability to troubleshoot distributed data processing environments.
• Strong communication skills and ability to collaborate with different technical teams.
• Availability to work in a hybrid model, 3 days per week in Tagus Park.
• Fluent Portuguese and English, both written and spoken.

Responsabilidades

• Developing and maintaining Change Data Capture (CDC) applications and data ingestion processes in Databricks.
• Creating and maintaining Databricks notebooks for setup, configuration, and data processing.
• Developing and parametrizing batch processing pipelines using Apache Spark.
• Implementing CDC logic to correctly detect and process inserts, updates, and deletes.
• Ensuring the generation of complete data snapshots while maintaining historical changes and data traceability.
• Working with Delta Lake to support reliable, scalable, and performant data processing.
• Working with multiple data sources, including Azure File Share, DB2, SQL Server, and SingleStore.
• Integrating processed data with destinations such as SingleStore and Kafka Confluent.
• Working with Databricks Workflows and Azure services to orchestrate and operationalize data pipelines.
• Optimizing processing performance and ensuring that pipelines are scalable and resilient.
• Contributing to the definition and evolution of data ingestion architecture and processing frameworks.
• Collaborating with other Data Engineers and technical teams to integrate processed data with downstream microservices.
• Ensuring data integrity, consistency, and traceability throughout migration and processing flows.
• Applying best practices around ETL/ELT, data engineering, monitoring, error handling, and pipeline reliability.
• Contributing to continuous improvement initiatives across the data platform and its ingestion frameworks.

Conhecimentos

Databricks

Apache Spark

Python

SQL

Delta Lake

CDC concepts

Azure

Kafka

Data ingestion

Ferramentas

Databricks notebooks

Azure services

Kafka Confluent

Descrição da oferta de emprego

At AdvanceWorks, we learn together, grow together, and have fun together. And we do all this while creating great software solutions for clients across many industries and geographies.

We are constantly challenging ourselves and each other to use the latest technologies and the best methodologies and make sure we walk the talk.

What will be your role

We are looking for a Data Architect to join our team in Lisbon and contribute to the development and evolution of modern data ingestion and Change Data Capture (CDC) solutions for a global organization operating at scale.

In this role, you’ll work on a modern data architecture built around Databricks, Apache Spark, Delta Lake, Azure, and Kafka, helping evolve a flexible and scalable approach to data ingestion across multiple source systems.

You’ll have an active role in defining and evolving ingestion patterns, developing Databricks notebooks and processing frameworks, configuring pipelines, and ensuring the integrity, traceability, and scalability of data throughout the process.

The project is moving from a traditional CDC approach focused primarily on Mainframe environments towards a more flexible architecture capable of supporting multiple data sources and destinations. This creates an opportunity to work on a technically challenging environment where Big Data, Cloud, CDC, and modern data engineering practices come together.

Your responsibilities will include

• Developing and maintaining Change Data Capture (CDC) applications and data ingestion processes in Databricks
• Creating and maintaining Databricks notebooks for setup, configuration, and data processing
• Developing and parametrizing batch processing pipelines using Apache Spark
• Implementing CDC logic to correctly detect and process inserts, updates, and deletes
• Ensuring the generation of complete data snapshots while maintaining historical changes and data traceability
• Working with Delta Lake to support reliable, scalable, and performant data processing
• Working with multiple data sources, including Azure File Share, DB2, SQL Server, and SingleStore
• Integrating processed data with destinations such as SingleStore and Kafka Confluent
• Working with Databricks Workflows and Azure services to orchestrate and operationalize data pipelines
• Optimizing processing performance and ensuring that pipelines are scalable and resilient
• Contributing to the definition and evolution of data ingestion architecture and processing frameworks
• Collaborating with other Data Engineers and technical teams to integrate processed data with downstream microservices
• Ensuring data integrity, consistency, and traceability throughout migration and processing flows
• Applying best practices around ETL/ELT, data engineering, monitoring, error handling, and pipeline reliability
• Contributing to continuous improvement initiatives across the data platform and its ingestion frameworks

What should you bring to the team

• 3+ years of experience in Data Engineering, Databricks, or related data-intensive roles
• Solid hands-on experience with Databricks, including notebooks, clusters, jobs, and workflows
• Strong knowledge of Apache Spark, including batch processing and distributed data transformations
• Strong experience with Python and SQL
• Hands-on experience with Delta Lake and modern lakehouse architectures
• Practical understanding of Change Data Capture (CDC) concepts and implementation patterns
• Experience developing and parametrizing data pipelines and processing frameworks
• Experience working with cloud data platforms, particularly Microsoft Azure
• Familiarity with Azure services such as Azure File Share and Data Lake
• Experience integrating data pipelines with Kafka, ideally Kafka Confluent
• Good understanding of data architecture and ETL/ELT best practices
• Experience designing scalable and resilient data processing solutions
• Strong problem-solving skills and ability to troubleshoot distributed data processing environments
• Strong communication skills and ability to collaborate with different technical teams
• Availability to work in a hybrid model, 3 days per week in Tagus Park
• Fluent Portuguese and English, both written and spoken

Nice to have

• Experience with SingleStore
• Experience with SQL-based CDC implementations across different database technologies
• Experience with file-based CDC or ingestion frameworks
• Knowledge of microservices and event-driven architectures
• Experience with MLOps or Data Engineering in cloud environments
• Familiarity with AI/GenAI pipelines and data platforms supporting AI initiatives
• Experience with streaming technologies and event-driven data architectures
• Knowledge of performance optimization techniques for Spark and Databricks
• Experience designing resilient and highly scalable data solutions

What is in it for you

• An amazing informal culture of smart, hardworking, and friendly people who support and care about each other
• A mentorship program from day one
• Opportunity to work on an innovative project combining CDC, Big Data, Cloud, Databricks, and Kafka
• The chance to contribute to the evolution of a modern and scalable data architecture
• Formal training and certifications in data, cloud, and engineering excellence
• Access to cutting-edge tools and technologies
• A dynamic company where your ideas matter more than job titles
• Flexibility with responsibility
• A Rubber Duck to help you debug those tricky Spark transformations

If you have any questions, drop us a line at [e-mail]