Data Engineer: PySpark, Databricks & AWS (Energy)

Yechte; Brussel; Sur place; EUR 60 000 - 90 000; Plein temps

Résumé du poste

Yechte is seeking a data engineer to develop and maintain scalable data solutions within our data framework. You will build pipelines using Python and PySpark, run ETL tasks in Databricks, and manage Spark clusters.

The role involves SQL optimisation, working with data warehouses, and ensuring adherence to technical standards. You will write technical specifications and unit tests, participate in architecture discussions, and implement data management best practices.

Qualifications

• Proven experience as a Data Engineer or in a similar data development role.

Responsabilités

• Develop and maintain data framework solutions.
• Design scalable data processing and transformation solutions.
• Develop data pipelines using Python and PySpark.
• Build and execute ETL processes using Databricks.
• Work with Spark and manage Spark clusters.
• Optimize SQL queries for data processing efficiency.
• Work with relational databases and data warehouses.
• Write technical specifications and unit tests.
• Participate in architecture discussions and contribute to design.
• Apply data management best practices and standards.
• Ensure timely, quality deliveries and production monitoring.
• Analyze and resolve production issues affecting data solutions.
• Collaborate with multidisciplinary teams and stakeholders.
• Translate customer requirements into effective data solutions.
• Contribute to continuous improvement of data engineering processes.

Connaissances

Python

PySpark

Databricks

AWS

SQL

Relational Databases

Data Warehouses

ETL Development

Unit Tests

Technical Specs

Architecture

Data Management

Communication

Outils

Apache Airflow

Azure DevOps

Terraform

Postman

CI/CD

C#

Description du poste

Yechte is seeking a data engineer to develop and maintain scalable data solutions within our data framework. You will build pipelines using Python and PySpark, run ETL tasks in Databricks, and manage Spark clusters.

The role involves SQL optimisation, working with data warehouses, and ensuring adherence to technical standards. You will write technical specifications and unit tests, participate in architecture discussions, and implement data management best practices.