Data Engineer - Azure Data Marketplace Platform (Hybrid)

BELVEDERE; Warszawa; Hybrid; PLN 180,000 - 280,000; Full time

Job summary

Belvedere is seeking a Data Engineer at Manager level to own data ingestion pipelines powering an Azure-based Data Marketplace. You will build scalable ETL processes and collaborate with cross-functional teams to deliver robust data assets.

Experience with Azure Data Factory, Azure Databricks, PySpark, Python and SQL is required; you will design reliable pipelines, ensure high availability, and contribute to innovative data products.

Qualifications

• 8-10 years of experience as a Data Engineer.
• Expert knowledge of Azure Data Factory, including pipeline development and management.
• Proficiency in Azure Databricks for data processing with notebooks, jobs and clusters.
• Strong programming skills in Python, PySpark and SQL.
• Experience with Apache Spark for big data processing.
• Solid understanding of ETL processes and data transformation.
• Familiarity with Azure storage solutions including Azure SQL Database, Blob Storage and Data Lake Storage.
• Knowledge of data warehousing principles and scalable data modelling techniques.
• Experience integrating multiple data sources in cloud environments.
• Ability to write modular, testable code for production systems.

Responsibilities

• Identify opportunities to improve data assets and services within the AI Data Marketplace
• Design and develop data ingestion pipelines with focus on stability and optimisation
• Build, test and maintain robust data pipelines and processes
• Write resilient, fault-tolerant and modular code for high availability systems
• Research and implement new methods to enhance data products
• Collaborate across teams to ensure smooth transitions through ingestion and consumption phases
• Manage Azure Data Factory pipelines, activities, triggers and data flows
• Process large datasets using Azure Databricks, Apache Spark and PySpark
• Transform and integrate data from various sources using ETL processes

Skills

Azure Data Factory

Azure Databricks

PySpark

Python

SQL

Apache Spark

ETL

Data Warehousing

Data modelling

Data ingestion pipelines

Tools

Azure Databricks notebooks

Azure Databricks jobs

Azure Databricks clusters

Azure SQL Database

Blob Storage

Data Lake Storage

Job description

Belvedere is seeking a Data Engineer at Manager level to own data ingestion pipelines powering an Azure-based Data Marketplace. You will build scalable ETL processes and collaborate with cross-functional teams to deliver robust data assets.

Experience with Azure Data Factory, Azure Databricks, PySpark, Python and SQL is required; you will design reliable pipelines, ensure high availability, and contribute to innovative data products.