Data Engineer

Yechte; Brussel; Sur place; EUR 60 000 - 90 000; Plein temps

Résumé du poste

Yechte is seeking a data engineer to develop and maintain scalable data solutions within our data framework. You will build pipelines using Python and PySpark, run ETL tasks in Databricks, and manage Spark clusters.

The role involves SQL optimisation, working with data warehouses, and ensuring adherence to technical standards. You will write technical specifications and unit tests, participate in architecture discussions, and implement data management best practices.

Qualifications

• Proven experience as a Data Engineer or in a similar data development role.

Responsabilités

• Develop and maintain data framework solutions.
• Design scalable data processing and transformation solutions.
• Develop data pipelines using Python and PySpark.
• Build and execute ETL processes using Databricks.
• Work with Spark and manage Spark clusters.
• Optimize SQL queries for data processing efficiency.
• Work with relational databases and data warehouses.
• Write technical specifications and unit tests.
• Participate in architecture discussions and contribute to design.
• Apply data management best practices and standards.
• Ensure timely, quality deliveries and production monitoring.
• Analyze and resolve production issues affecting data solutions.
• Collaborate with multidisciplinary teams and stakeholders.
• Translate customer requirements into effective data solutions.
• Contribute to continuous improvement of data engineering processes.

Connaissances

Python

PySpark

Databricks

AWS

SQL

Relational Databases

Data Warehouses

ETL Development

Unit Tests

Technical Specs

Architecture

Data Management

Communication

Outils

Apache Airflow

Azure DevOps

Terraform

Postman

CI/CD

C#

Description du poste

Job Responsibilities

Develop and maintain solutions within the data framework. Design and implement scalable data processing and transformation solutions. Develop data pipelines using Python and PySpark. Build and execute ETL processes using Databricks. Work with Apache Spark and manage Spark clusters. Develop and optimise SQL queries for efficient data processing. Work with relational databases and data warehouses. Write and maintain technical specifications. Write and execute unit tests to ensure solution quality. Participate in architecture discussions and contribute to technical solution design. Apply and implement best practices around data management. Ensure data solutions comply with agreed technical and architectural standards. Ensure deliveries are completed within agreed timelines and quality standards. Perform production monitoring and maintenance. Analyse and resolve production issues affecting data solutions. Collaborate with multidisciplinary teams and stakeholders. Understand customer requirements and translate them into effective technical solutions. Contribute to the continuous improvement of data engineering processes and solutions.

Must Have Skills

Proven experience as a Data Engineer or in a similar data development role. Strong hands-on experience with Python V3. Strong experience with PySpark. Strong experience with Databricks. Strong experience with AWS. Solid understanding of: SQL Relational databases Data warehouses Experience with ETL development and data transformation. Experience with Databricks, including: Creating and executing ETL tasks. Managing Spark clusters. Optimising SQL queries. Experience writing technical specifications. Experience writing unit tests. Good understanding of data management best practices. Ability to participate in architecture and solution design activities. Strong analytical and problem-solving skills. Ability to work independently and proactively. Strong communication and interpersonal skills. Ability to work effectively in a multidisciplinary environment. Ability to understand customer requirements and deliver appropriate data solutions. Experience with production monitoring and maintenance.

Nice to have

Experience with Apache Airflow. Experience with Azure DevOps / TFS / VSTS. Experience with Terraform. Experience with Insomnia / Postman. Knowledge of CI/CD practices. Knowledge of C#. Knowledge of the energy market industry. Knowledge of portfolio management. Knowledge of risk management. Knowledge of Trading. Knowledge of energy forecasting activities. Experience with Web APIs. Knowledge of Agile methodology. Experience within the energy trading or energy forecasting domain.

What\'s great in the job?

Work on strategic data engineering initiatives within the energy sector. Contribute to the industrialisation and securing of critical business processes. Work with modern technologies including Python, PySpark, Databricks, AWS, and Apache Spark. Contribute to scalable and reliable data solutions supporting gas, power, costing, exposure management, and forecasting. Collaborate with multidisciplinary teams across Data, IT, Architecture, and Business. Participate in architecture discussions and contribute to data engineering best practices. Work in a dynamic and fast-paced environment focused on innovation and continuous improvement. Opportunity to contribute to the evolution of the Downstream IT application landscape.