Azure Data Engineer

Ubique Systems; Amsterdam; On-site; EUR 70,000 - 100,000; Full time

Job summary

Ubique Systems in Amsterdam is seeking a data engineer to design, build and maintain data pipelines in Azure cloud. You will work with PySpark, Databricks and Python to deliver scalable ETL solutions and optimize performance.

You will collaborate with cross-functional teams, implement CI/CD with Azure DevOps, and ensure data security and quality according to standards. Prior experience with Azure Data Factory is essential.

Qualifications

• Experience with Azure Data Factory to orchestrate data movement and transformation.
• Proficiency in Python and PySpark for big data processing.
• Hands-on with Azure Databricks for building scalable data pipelines.
• Familiarity with SQL for querying and managing large datasets.

Responsibilities

• Design, develop, and maintain complex data pipelines using Azure Data Factory and Azure Databricks.
• Build and optimize large-scale data processing solutions using PySpark and Databricks.
• Collaborate with cross-functional teams to understand data requirements and deliver efficient data solutions.
• Implement CI/CD pipelines in Azure DevOps for automated testing, integration, and deployment of data solutions.
• Develop, test, and maintain robust data solutions in Python for ETL processes.
• Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
• Ensure data security and compliance with company standards and best practices.
• Stay updated with the latest trends and best practices in cloud data engineering and Azure services.

Skills

Azure Data Factory

Azure Databricks

Python

PySpark

Azure DevOps

CI/CD

SQL

Job description

Must have skills –.

Key Responsibilities:

• Design, develop, and maintain complex data pipelines using Azure Data Factory and Azure Databricks.
• Build and optimize large-scale data processing solutions using PySpark and Databricks.
• Collaborate with cross-functional teams to understand data requirements and deliver efficient data solutions.
• Implement CI/CD pipelines in Azure DevOps for automated testing, integration, and deployment of data solutions.
• Develop, test, and maintain robust data solutions in Python for data extraction, transformation, and loading (ETL) processes.
• Monitor, troubleshoot, and optimize data pipelines for performance and reliability
• Ensure data security and compliance with company standards and best practices.
• Stay updated with the latest trends and best practices in cloud data engineering and Azure services.

Required Skills and Qualifications:

• Experience with Azure Data Factory for orchestrating and automating data movement and transformation.
• Proficiency in Azure Databricks for building and managing scalable data processing frameworks.
• Strong programming skills in Python and experience working with PySpark for big data processing.
• Knowledge of Azure DevOps and experience with CI/CD pipeline implementation.
• Familiarity with SQL for querying and managing large datasets.
• Strong problem-solving skills and the ability to work independently or as part of a team.