Azure Data Engineer – Azure Databricks & PySpark

Ubique Systems; Amsterdam; On-site; EUR 65,000 - 90,000; Full time

Job summary

Ubique Systems in Amsterdam is seeking a data engineer to design, build, and maintain scalable data pipelines using Azure Data Factory and Databricks. You will implement PySpark-based processing and Python ETL workflows.

You will collaborate across teams, implement CI/CD pipelines with Azure DevOps, ensure data security, and stay current with cloud data engineering trends.

Qualifications

• Experience designing and maintaining complex data pipelines.
• Proficient in PySpark and Databricks for large-scale processing.
• Experience with Azure Data Factory orchestrating ETL workflows.
• Strong Python programming for ETL processes.
• Knowledge of SQL for querying large datasets.
• Familiarity with CI/CD pipelines in Azure DevOps.
• Focus on data security and compliance.
• Ability to collaborate with cross-functional teams.
• Keep up-to-date with cloud data engineering trends.

Responsibilities

• Design, develop, and maintain data pipelines.
• Build and optimize large-scale data processing solutions.
• Collaborate with cross-functional teams to translate data requirements.
• Implement CI/CD pipelines for automated testing and deployment.
• Develop and maintain Python ETL processes.
• Monitor and optimize pipeline performance and reliability.
• Ensure data security and compliance with company standards.

Skills

Azure Data Factory

Azure Databricks

PySpark

Python

CI/CD

Azure DevOps

SQL

Tools

Azure Data Factory

Azure Databricks

Azure DevOps

PySpark

Job description

Required Skills and Qualifications:

• Design, develop, and maintain complex data pipelines using Azure Data Factory and Azure Databricks.


• Build and optimize large-scale data processing solutions using PySpark and Databricks.


• Collaborate with cross-functional teams to understand data requirements and deliver efficient data solutions.


• Implement CI/CD pipelines in Azure DevOps for automated testing, integration, and deployment of data solutions.


• Develop, test, and maintain robust data solutions in Python for data extraction, transformation, and loading (ETL) processes.


• Monitor, troubleshoot, and optimize data pipelines for performance and reliability.


• Ensure data security and compliance with company standards and best practices.


• Stay updated with the latest trends and best practices in cloud data engineering and Azure services.


• Experience with Azure Data Factory for orchestrating and automating data movement and transformation.


• Proficiency in Azure Databricks for building and managing scalable data processing frameworks.


• Strong programming skills in Python and experience working with PySpark for big data processing


• Knowledge of Azure DevOps and experience with CI/CD pipeline implementation.


• Familiarity with SQL for querying and managing large datasets.