Data engineer in capital markets

Autoriteit Financiële Markten; Amsterdam; On-site; EUR 64,000 - 106,000; Full time

Benefits offered by this job

Laptop и смартфон

Обучение и развитие

Компенсация расходов на транспорт

Рабочее место дома

Job summary

Autoriteit Financiële Markten ищет Data Engineer для проектирования и поддержки конвейеров данных в Azure и Databricks. Вы будете работать над сбором, обработкой и подготовкой данных для анализа и надмандитного мониторинга рынков капитала.

Требуется опыт работы с Databricks, PySpark, SQL и Azure Data Factory; знание CI/CD, DevOps и архитектуры данных. Предлагается годовой контракт с возможной переработкой в штат и конкурентным уровнем оплаты.

Qualifications

• Опыт работы с Databricks, PySpark и SQL в Azure.
• Разработка и управление масштабируемыми пайплайнами данных.

Responsibilities

• Разработка и управление процессами загрузки, трансформации и обработки данных в Azure и Databricks.
• Настройка CI/CD и автоматизированного тестирования и развёртывания.
• Работа с MLOps, унифицированными каталогами и обеспечением качества данных.
• Сотрудничество с аналитиками и инженерами для построения устойчивых решений.

Skills

Databricks

PySpark

SQL

Azure Data Factory

CI/CD

Azure DevOps

Unity Catalog

MLOps

Data modeling

Data governance

Git

Education

Master's degree in Computer Science / Data Engineering

Tools

Databricks

Azure Data Factory

Azure Storage

Declarative Automation Bundles

Unity Catalog

Git

Azure DevOps

Job description

Описание

As an all-round Data Engineer within the Data-Driven Capital Markets department (DAK), you design, build and manage reliable data pipelines and datasets for the supervision of capital markets. You also help further professionalize our development and deployment processes and the use of Databricks. You make data accessible, structure it and prepare it for data analysts, supervisors and other users.

Задачи

Developing and managing data ingestion, transformation and processing processes within our Azure cloud environment and on the Databricks platform, using, among other things, Databricks Jobs, Azure Data Factory and PySpark/SQL. Setting up and improving automated development, testing and deployment processes with CI/CD in Azure DevOps. Recording and controlled deployment of Databricks code, configurations and resources with Declarative Automation Bundles. Developing and managing data pipelines and deployment processes for machine learning applications and contributing to the further professionalization of MLOps within the Databricks platform. Applying frameworks and standards for data quality, security, privacy and governance, including the use of Unity Catalog. Monitoring and improving the operation and reliability of data pipelines and contributing to cost-conscious use of the Databricks platform. Actively contributing to architecture questions around scalability, security, data governance and cloud configuration. Working with data analysts, data management specialists and platform engineers to translate technical and functional questions into future-proof solutions. Identifying opportunities for improvement and actively contributing to innovation, standardization and knowledge sharing within the team.

Требования

Master's-level working and thinking ability and at least three years of relevant work experience as a Data Engineer or in a comparable position within an Azure cloud environment. Data Engineering: Extensive experience with Databricks, PySpark and SQL, and with developing and managing scalable data ingestion, transformation and processing processes. Azure: Experience with Azure Data Factory, Azure Storage and other relevant Azure services for data processing. DevOps and automation: Experience with Git, Azure DevOps and CI/CD, and an understanding of controlled, repeatable and automated testing and deployment of data solutions. Databricks development: Experience with or knowledge of Declarative Automation Bundles and Unity Catalog. MLOps: Experience developing data pipelines, reusable components or deployment processes for machine learning applications, or the motivation to develop further in this area. Security and governance: You can apply frameworks for identity and access management, data security, privacy and governance when developing data solutions. Monitoring: Experience monitoring data pipelines, analyzing disruptions and improving performance and reliability. Data modeling: Knowledge of relational and dimensional data modeling, including normalization and the principles of Inmon and Kimball. Way of working: You work iteratively, take ownership and actively seek collaboration with colleagues and other disciplines. Personal qualities: You are analytical, enterprising and focused on developing sustainable solutions. Будет плюсом: Relevant certifications in Databricks, Azure Data Engineering or DevOps; Experience with ML Engineering or AI Engineering in an enterprise environment

Условия

You start with a one-year contract, with the prospect of permanent employment. Placement in salary scale G, with a gross annual salary between €64,000 and €106,000 (36 hours, including 13th-month pay and holiday allowance), depending on knowledge and experience. A 36-hour working week, with working days agreed by mutual consent (for example, 4x9). 199.5 vacation hours per year (in addition to your standard rostered day off).

• A laptop and a smartphone.


• Extensive training and development opportunities.


• Full reimbursement of public transport travel costs (2nd class).


• A work-from-home allowance for days worked from home.


• A contribution towards a gym membership.


• Attractive secondary benefits such as a bicycle and device scheme and reimbursement for your home workspace.