Data Engineer

Inetum; Lisboa; Presencial; EUR 40 000 - 80 000; Tempo integral

Resumo da oferta

Join a forward-thinking company as a Data Engineer, where you'll build and maintain robust data infrastructure. This role involves creating data pipelines with Airflow, optimizing code, and collaborating with analysts to deliver reliable data solutions. You'll work in a dynamic environment focused on digital transformation, leveraging advanced technologies like Hadoop, Spark, and Docker. If you're passionate about data and eager to make a significant impact, this opportunity is perfect for you!

Qualificações

• Experience with Data Warehouses and Data Lakes management.
• Strong knowledge of the Hadoop ecosystem and Spark architecture.

Responsabilidades

• Maintain and optimize data pipelines using Airflow.
• Collaborate with data analysts to ensure data accessibility.

Conhecimentos

Advanced Python

API creation in Python (Flask)

SQL skills

Hadoop ecosystem (HDFS + Yarn)

Apache Airflow

Containerization with Docker

Kubernetes

Apache Kafka

Automation with Jenkins

Agile methodologies (SCRUM)

Ferramentas

Docker

Kubernetes

Apache Airflow

Jenkins

Ansible

Descrição da oferta de emprego

4 days ago Be among the first 25 applicants

Get AI-powered advice on this job and more exclusive features.

Company Description

Inetum is a European leader in digital services. Inetum’s team of 28,000 consultants and specialists strive every day to make a digital impact for businesses, public sector entities, and society. Inetum’s solutions aim at contributing to its clients’ performance and innovation as well as the common good. Present in 19 countries with a dense network of sites, Inetum partners with major software publishers to meet the challenges of digital transformation with proximity and flexibility. Driven by its ambition for growth and scale, Inetum generated sales of 2.5 billion euros in 2023.

Job Description

We are looking for a Data Engineer responsible for building and maintaining the infrastructure supporting the organization’s data architecture. The role involves creating and managing data pipelines using Airflow for data extraction, processing, and loading, ensuring their maintenance, monitoring, and stability. The engineer will work closely with data analysts and end-users to provide accessible and reliable data.

Main Tasks and Responsibilities

• Maintain the infrastructure supporting the current data architecture.
• Create data pipelines in Airflow for data extraction, processing, and loading.
• Maintain, monitor, and ensure the stability of data pipelines.
• Provide data access to data analysts and end-users.
• Manage DevOps infrastructure.
• Deploy Airflow DAGs to production using DevOps tools.
• Optimize code and queries.
• Review code.
• Improve current data architecture and DevOps processes.
• Deliver data in useful and appealing ways to users.
• Perform, document, and review analysis on regulatory topics.
• Understand business changes and requirements, assess impact and costs.

Qualifications

Technical Skills:

• Advanced Python (Mandatory)
• Experience creating APIs in Python, especially Flask (Mandatory)
• Experience in documenting and testing in Python (Mandatory)
• Advanced SQL skills and relational database management (Oracle mandatory, SQL Server and PostgreSQL desirable)
• Experience with Data Warehouses
• Hadoop ecosystem - HDFS + Yarn (Mandatory)
• Spark Environment Architecture (Mandatory)
• Advanced PySpark (Mandatory)
• Experience in distributed environments using Hadoop and Spark
• Data Lakes management (Mandatory), S3 preferred
• Experience with Parquet format (Mandatory), Avro a plus
• Apache Airflow experience in pipeline development and deployment (Mandatory)
• Containerization with Docker (Mandatory)
• Kubernetes (Mandatory)
• Apache Kafka (Mandatory)
• Automation of deployment using Jenkins (Mandatory), Ansible a plus
• Agile methodologies, at least SCRUM

Language Skills:

• Fluent in English (Mandatory)

Seniority level

• Mid-Senior level

Employment type

• Full-time

Job function

• Analyst

Industries

• IT Services and IT Consulting