Senior Data Engineer

AgnesCole Consulting Services Pvt Ltd; Pays-Bas; Sur place; EUR 70 000 - 95 000; Plein temps

Résumé du poste

AgnesCole Consulting Services Pvt Ltd seeks a Senior Data Engineer to design, build and optimise scalable data pipelines and cloud-based data platforms for energy and renewables data. You will work with Python, SQL, Azure Databricks and Spark to process IoT, operational and real-time energy data and ensure data quality and security across pipelines.

Ideal candidates have hands-on Py/SQL, Delta Lake, and Azure experience, plus strong collaboration with architects, analysts and engineering teams

Qualifications

• Strong hands-on experience with Python and SQL.
• Advanced expertise in Azure Databricks and Apache Spark.
• Proven experience developing real-time and batch data pipelines.
• Strong knowledge of Delta Lake and modern data platform architecture.

Responsabilités

• Design, build and maintain scalable data pipelines using Python, SQL, Azure Databricks and Apache Spark.
• Develop real-time and batch data processing solutions for energy and industrial IoT data.
• Implement and optimise Delta Lake architecture for efficient data storage and processing.
• Design and maintain cloud-based data platforms within Microsoft Azure.
• Integrate data from IoT sensors, operational systems and external data sources.
• Ensure data quality, reliability, performance and security across data pipelines.
• Collaborate with data architects, analysts and engineering teams to deliver robust data solutions.
• Support performance optimisation, troubleshooting and continuous improvement of data infrastructure.

Connaissances

Python

SQL

Azure Databricks

Apache Spark

Real-time processing

Data modelling

Outils

Delta Lake

Description du poste

We are seeking an experienced Senior Data Engineer to support a data engineering programme within the energy and renewables sector. The successful candidate will design, develop and optimise scalable data pipelines and cloud-based data platforms, supporting the processing of large-scale IoT, operational and real-time energy data.

Key Responsibilities

• Design, build and maintain scalable data pipelines using Python, SQL, Azure Databricks and Apache Spark.
• Develop real-time and batch data processing solutions for energy and industrial IoT data.
• Implement and optimise Delta Lake architecture for efficient data storage and processing.
• Design and maintain cloud-based data platforms within Microsoft Azure.
• Integrate data from IoT sensors, operational systems and external data sources.
• Ensure data quality, reliability, performance and security across data pipelines.
• Collaborate with data architects, analysts and engineering teams to deliver robust data solutions.
• Support performance optimisation, troubleshooting and continuous improvement of data infrastructure.

Required Experience

• Strong hands-on experience with Python and SQL.
• Advanced expertise in Azure Databricks and Apache Spark.
• Proven experience developing real-time and batch data pipelines.
• Strong knowledge of Delta Lake and modern data platform architecture.
• Experience working with IoT data, streaming technologies and large-scale datasets.
• Understanding of Azure data services and cloud-based data engineering.
• Experience with data modelling, ETL/ELT processes and pipeline optimisation.
• Strong problem-solving and stakeholder communication skills.

Preferred Qualifications

• Previous experience within energy, utilities, renewable energy or industrial IoT environments.
• Experience with Azure Event Hubs, Azure Data Factory or Structured Streaming.
• Familiarity with time-series data, sensor telemetry and operational analytics.
• Knowledge of CI/CD, infrastructure automation and data platform governance.