Senior Data Engineer: Pipelines on Databricks & AWS

PRACYVA; Brussel; Sur place; EUR 90 000 - 130 000; Plein temps

Résumé du poste

PRACYVA in Brussels is seeking a Senior Data Engineer to design and implement end-to-end data pipelines using Python, PySpark, Databricks, AWS, and SQL. You will collaborate with Data Analysts, Data Scientists, IT teams, and business stakeholders to deliver reliable data products in a scalable, enterprise environment.

You will own ETL/data processing pipelines, Data Lake management, data quality, and production stability, optimize Spark and SQL queries, create Databricks ETL jobs and Spark

Qualifications

• Strong experience with Python 3 and PySpark.
• Extensive Databricks and AWS expertise.
• Proficient SQL and data warehousing skills.
• Experience in building ETL pipelines and data lakes.
• Familiarity with Agile methodologies.

Responsabilités

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Connaissances

Python 3

PySpark

Databricks

AWS

SQL / Relational Databases / Data Ware

Outils

Apache Airflow

Azure DevOps

Terraform

Postman

Insomnia

Description du poste

PRACYVA in Brussels is seeking a Senior Data Engineer to design and implement end-to-end data pipelines using Python, PySpark, Databricks, AWS, and SQL. You will collaborate with Data Analysts, Data Scientists, IT teams, and business stakeholders to deliver reliable data products in a scalable, enterprise environment.

You will own ETL/data processing pipelines, Data Lake management, data quality, and production stability, optimize Spark and SQL queries, create Databricks ETL jobs and Spark