Hybrid Data Engineer: Azure, Databricks & Pipelines

Capgemini; Vila Nova de Gaia; Hybrid; EUR 55,000 - 90,000; Full time

Benefits offered by this job

Health insurance

Hybrid work policy

Job summary

Capgemini Portugal is building a new Tech Hub in Porto and invites a Data Engineer to join a collaborative, hybrid environment. You will design and maintain batch, streaming, and CDC data pipelines using SAP, Kafka, APIs, and databases, delivering trusted data for analytics and AI products.

You will work with Azure Data Factory, Databricks, Spark, SQL, and Delta Lake, focusing on data quality, monitoring, and scalable architectures, while collaborating across DataOps, AI, and cross-functional

Qualifications

• Proficiency in Python or Scala and SQL for production data pipelines.
• Practical experience with Spark, Delta Lake and Databricks, including basic performance optimisation.
• Experience with batch pipelines and at least one of change data capture or streaming patterns.
• Practical experience with Azure data services, including Azure Data Factory.
• Experience with automated data-quality testing and pipeline monitoring.
• Working knowledge of Git, CI/CD and code review.
• Working knowledge of Unity Catalog or comparable data-governance and access-control tooling.
• Confluent Kafka, Spark Structured Streaming or comparable streaming frameworks (nice to have).
• SAP S/4HANA, BW and Business Data Cloud extraction and data structures (nice to have).
• Document processing, embeddings and vector indexes for AI retrieval (nice to have).

Responsibilities

• Build and maintain batch, CDC, and streaming data pipelines across SAP, Kafka, APIs, and databases.
• Develop scalable data solutions using Azure Data Factory, Databricks, Spark, SQL, and Delta Lake.
• Implement data quality, monitoring, observability, alerting, and data contract controls.
• Optimize pipeline performance, cost efficiency, and security through Unity Catalog.
• Contribute reusable frameworks, CI/CD processes, and data engineering best practices.
• Collaborate with DataOps, ML Engineering, AI, Analytics Engineers, and cross-functional teams.
• Deliver trusted, curated data to support analytics and AI products.
• Participate in code reviews and support continuous improvement initiatives.
• Good understanding of end-to-end data engineering, data quality, and data lifecycle management.
• Experience working with data contracts, stakeholder collaboration, and analytics-driven use cases.

Skills

Python

SQL

Spark

Delta Lake

Databricks

Azure Data Factory

CI/CD

Git

Unity Catalog

Confluent Kafka

Tools

Databricks

Delta Lake

Spark Structured Streaming

Confluent Kafka

Job description

Capgemini Portugal is building a new Tech Hub in Porto and invites a Data Engineer to join a collaborative, hybrid environment. You will design and maintain batch, streaming, and CDC data pipelines using SAP, Kafka, APIs, and databases, delivering trusted data for analytics and AI products.

You will work with Azure Data Factory, Databricks, Spark, SQL, and Delta Lake, focusing on data quality, monitoring, and scalable architectures, while collaborating across DataOps, AI, and cross-functional