Big Data Engineer

act digital; Porto; Presencial; EUR 45 000 - 65 000; Tempo integral

Resumo da oferta

act digital in Porto is looking for an experienced Data Engineer/BI Engineer to design and operate Data Lake pipelines using Python and PySpark, delivering scalable data solutions for analytics. You will collaborate with Data Engineers, BI Developers, Analysts, and stakeholders to ensure robust data delivery and performance optimization.

Strong SQL and data modeling foundations are essential, along with hands-on experience in Big Data, NoSQL, and DevOps practices.

Qualificações

• 5+ years of professional experience as a Data Engineer / BI Engineer.
• Minimum 4 years of hands-on experience with Big Data and NoSQL technologies (Hadoop, HDFS, Hive, Kerberos, Starburst).
• Experience developing Data Lake pipelines using Python and PySpark (Spark is a plus).
• Solid knowledge of SQL and Data Modeling principles.
• Experience with Visual Studio Code or similar IDEs.
• Familiarity with DevOps concepts and CI/CD practices.
• Experience with Git for source code management.
• Experience with Jenkins for build automation.
• Experience with XL Deploy and XL Release for deployment processes.
• English level B2 or higher (mandatory).

Responsabilidades

• Develop and maintain Data Lake pipelines using Python and PySpark.
• Analyze business and technical requirements and propose scalable Data Lake architecture solutions.
• Gather, document and manage application requirements within an Agile environment.
• Design and implement data solutions ranging from simple to complex use cases, promoting automation and operational efficiency.
• Maintain and enhance existing data processes while providing technical support to users.
• Collaborate with Data Engineers, BI Developers, Data Analysts, Business Analysts, Software Developers and business stakeholders to deliver data-driven solutions.
• Contribute throughout the full project lifecycle, including assessment, use case identification, data preparation, implementation and industrialization.
• Identify opportunities to improve existing processes, with a strong focus on performance optimization.
• Work within a cross-functional international team across Porto and Paris, contributing to knowledge sharing and continuous improvement of the data ecosystem.

Conhecimentos

Data engineering

Python

PySpark

SQL

Data Modeling

English (B2)

Ferramentas

Hadoop

HDFS

Hive

Kerberos

Starburst

Git

Jenkins

VS Code

CI/CD

XL Deploy

XL Release

REST APIs

Power BI

SharePoint Online

Dataverse

Azure

Descrição da oferta de emprego

• 5+ years of professional experience as a Data Engineer / BI Engineer.
• Minimum 4 years of hands‑on experience with Big Data and NoSQL technologies (Hadoop, HDFS, Hive, Kerberos, Starburst).
• Experience developing Data Lake pipelines using Python and PySpark (Spark is a plus).
• Solid knowledge of SQL and Data Modeling principles.
• Experience with Visual Studio Code or similar IDEs.
• Familiarity with DevOps concepts and CI/CD practices.
• Experience with Git for source code management.
• Experience with Jenkins for build automation.
• Experience with XL Deploy and XL Release for deployment processes.
• English level B2 or higher (mandatory).

Nice to Have

• Experience with Spark optimization and performance tuning (Shell Scripting, Execution Plans).
• Knowledge of Power BI, Power Apps and Power Automate.
• Experience administering SharePoint Online and Dataverse environments.
• Experience with REST APIs.
• Knowledge of SSAS, SSIS and Azure.

Responsibilities

• Develop and maintain Data Lake pipelines using Python and PySpark.
• Analyze business and technical requirements and propose scalable Data Lake architecture solutions.
• Gather, document and manage application requirements within an Agile environment.
• Design and implement data solutions ranging from simple to complex use cases, promoting automation and operational efficiency.
• Maintain and enhance existing data processes while providing technical support to users.
• Collaborate with Data Engineers, BI Developers, Data Analysts, Business Analysts, Software Developers and business stakeholders to deliver data-driven solutions.
• Contribute throughout the full project lifecycle, including assessment, use case identification, data preparation, implementation and industrialization.
• Identify opportunities to improve existing processes, with a strong focus on performance optimization.
• Work within a cross-functional international team across Porto and Paris, contributing to knowledge sharing and continuous improvement of the data ecosystem.