Big Data Engineer Python

Europe; Brussel Hoofdstad; Presencial; EUR 65 000 - 95 000; Tempo integral

Resumo da oferta

La Fosse in Belgium is seeking a Big Data Engineer to develop, integrate, and optimise large-scale data flows in close collaboration with data scientists. You will design robust data pipelines using Hadoop, Spark, Hive, Airflow, and modern data technologies, performing batch ETL and API integration.

The role emphasizes hands-on data cleansing and orchestration, with experience in cloud architectures (AWS/Azure) and full-time availability. This position is based in Belgium.

Qualificações

• Experience with real-time architectures and event streaming infrastructure is advantageous.

Responsabilidades

• Collaborate with data scientists to develop effective technical solutions.
• Perform data cleansing, preparation and integration.
• Design and build robust data pipelines.
• Manage component orchestration.
• Handle batch Spark ETL, including wrangling, joining, and aggregation.

Conhecimentos

Collaboration

Analytical

Organised

Problem Solving

Data cleansing

Pipeline design

System integration

Ferramentas

Hadoop

Spark

Hive

Airflow

Docker

Kubernetes

Python

API development

AWS

Azure

Descrição da oferta de emprego

Big Data Engineer

This role focuses on developing, integrating, and optimising large-scale data flows. It involves close collaboration with data scientists to translate analytical models into practical technical solutions, requiring hands-on expertise in data cleansing, pipeline design, component orchestration, and modern data technologies such as Hadoop, Spark, and API development.

Mission & Responsibilities

• Technical Collaboration: work closely with data scientists to develop effective technical solutions
• Data Optimisation: perform data cleansing, preparation, and integration
• Pipeline Engineering: design and build robust data pipelines
• System Integration: manage component orchestration
• Large-Scale Data Processing: responsible for batch Spark ETL, including data wrangling, joining, and aggregation

Technical Competencies

• AI Ecosystems & Frameworks, including:
  • Modern Data Stack (MDS)
  • On-premises Data Lake technologies (Hadoop, Spark, Hive)
  • Large-scale data processing (Spark)
  • Orchestration tools (Apache Airflow)
  • Containerisation (Docker / Kubernetes)
  • Python programming
• Architecture & Integrations:
  • API development
  • Cloud architectures (AWS, Azure)

Experience with real-time architectures and event streaming infrastructure is advantageous.

Behavioural Competencies

• Collaboration: able to work closely and effectively with data scientists to translate complex analytical needs into functional technical solutions
• Analytical & Organised: precise and methodical approach to data cleansing, preparation, integration, and pipeline design
• Problem Solving: quick and efficient in addressing bottlenecks in data processing or orchestration errors

We require full-time availability (100%), although 80% availability will be considered.