Data engineer in customer lake

QiBit; Lisboa; Hybrid; EUR 45,000 - 55,000; Full time

Job summary

QiBit Portugal (Gi Group Holding) is seeking a Data Engineer to join a team building and evolving a data platform on Azure Databricks, impacting pipelines' quality, governance, and scalability. The role focuses on delivering robust data workflows and collaborating with cross-functional stakeholders in a hybrid Lisbon-based office.

You will develop, optimize ingestion pipelines, implement PySpark transformations, and contribute to governance through Unity Catalog, with opportunities for growth in

Qualifications

• Experience as a Data Engineer in cloud data environments.
• Hands-on experience with Azure Databricks and PySpark development.
• Experience in data ingestion (ETL/ELT) and building reusable pipelines.
• Knowledge of governance, catalogs, and access control, ideally with Unity Catalog.
• Strong collaboration and communication skills in a multidisciplinary team context.
• Nice to have: Databricks Apps; Unity AI Gateway; data lake design and data quality practices.

Responsibilities

• Develop and optimize data ingestion pipelines (batch/streaming) focusing on reliability and performance.
• Implement data transformations in PySpark on Azure Databricks.
• Contribute to governance and permissions through Unity Catalog.
• Support operationalization of Databricks Apps components and align with business needs.
• Collaborate with Data, Cloud, and Analytics teams to translate requirements into scalable solutions.
• Ensure quality, observability, and documentation of pipelines and datasets in a customer lake context.

Skills

Azure Databricks

PySpark

Data pipelines

Cloud data environments

Collaboration

Tools

Unity Catalog

Databricks Apps

Job description

Описание:

QiBit Portugal (Gi Group Holding) supports the recruitment of specialized talent in IT and Digital for transformation projects at leading clients. We are looking for a Data Engineer to join a team focused on building and evolving a data platform on Azure Databricks, with a direct impact on the quality, governance, and scalability of pipelines.

Задачи:

• Develop and optimize data ingestion pipelines (batch and/or streaming, depending on the context), focusing on reliability and performance
• Implement data transformations and processing in PySpark on Azure Databricks
• Contribute to implementing and maintaining governance and permissions through Unity Catalog
• Support the operationalization of applications and components in the Databricks Apps ecosystem, aligning deliverables with business needs
• Work closely with technical stakeholders (Data, Cloud, and Analytics) to translate requirements into scalable solutions
• Ensure good practices for the quality, observability, and documentation of pipelines and datasets in a Customer Lake context

Требования:

• Experience as a Data Engineer in cloud data environments
• Hands‑on experience with Azure Databricks and development in PySpark
• Experience in Data Ingestion (ETL/ELT) and building reusable pipelines
• Knowledge of governance, catalogs, and access control, ideally with Unity Catalog
• Strong collaboration and communication skills in a multidisciplinary team context
• Будет плюсом: Experience with Databricks Apps; Exposure to AI/serving components in the Databricks ecosystem, including Unity AI Gateway; Experience designing/organizing a data lake / customer lake and applying good practices for data modeling and quality

Условия:

• A project focused on modern data engineering and good governance practices in Databricks
• Prospects for continuity and technical growth in a results-oriented team
• Development and learning opportunities in a cloud and data platform context
• Permanent contract, hybrid working arrangement with 3 days per week in the office in Lisbon
• Transparent salary conditions Under Directive (EU) 2023/970, the starting base salary for this role is €50,000 gross per year