Data Engineer (ID: 3583)

Stafide; Amsterdam; On-site; EUR 70,000 - 90,000; Full time

Benefits offered by this job

Opportunity to work on large-scale cloud data platforms

Collaborative environment

Challenging projects

Job summary

A leading data solutions company in Amsterdam is seeking a Data Engineer with over 8 years of experience in architecting and implementing scalable data pipelines using Databricks and Azure Data Factory. The role involves optimizing data ingestion processes, ensuring data quality, and collaborating with cross-functional teams. Ideal candidates will possess advanced Python scripting skills and a strong understanding of data governance principles. This position offers an exciting opportunity to work on large-scale cloud data platforms.

Qualifications

• 8+ years of experience in data engineering or big data environments.
• Strong hands-on experience with Databricks and PySpark.
• Practical experience with Azure Data Factory for data pipeline development.
• Advanced Python scripting skills.
• Solid understanding of data governance principles.
• Experience implementing data quality frameworks.
• Hands-on experience with data lineage tracking.
• Experience with large-scale data processing frameworks.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, and Azure Data Factory.
• Develop and optimize data ingestion, transformation, and loading processes.
• Implement data governance frameworks.
• Build data quality validation processes.
• Establish data lineage tracking mechanisms.
• Develop and maintain Python scripts for data workflows.
• Collaborate with data scientists and analysts.
• Optimize data pipeline performance within Azure.

Skills

Databricks

PySpark

Azure Data Factory

Python scripting

Data governance

Data quality frameworks

Data lineage tracking

Job description

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, and Azure Data Factory.
• Develop and optimize data ingestion, transformation, and loading processes for structured and unstructured data sources.
• Implement data governance frameworks to ensure proper data ownership, compliance, and security standards.
• Build and maintain reliable data quality validation processes to ensure accuracy, consistency, and completeness of enterprise data.
• Establish data lineage tracking mechanisms to provide transparency on how data flows across systems and pipelines.
• Develop and maintain Python scripts and automation tools to streamline data engineering workflows.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality datasets for analytics and AI initiatives.
• Optimize data pipeline performance and scalability within Azure-based data platforms.

What You Bring to the Table

• 8+ years of experience in data engineering, data platform development, or big data environments.
• Strong hands‑on experience with Databricks and PySpark for distributed data processing.
• Practical experience working with Azure Data Factory for orchestration and data pipeline development.
• Advanced Python scripting skills for automation, transformation, and integration tasks.
• Solid understanding of data governance principles, including metadata management, data cataloging, and compliance.
• Experience implementing data quality frameworks and validation rules within data pipelines.
• Hands‑on experience with data lineage tracking and documentation practices.
• Experience working with large‑scale data processing frameworks and cloud‑based data platforms.

You Should Possess the Ability to

• Architect and implement end‑to‑end data pipelines that support large‑scale data processing.
• Design efficient data models and transformation logic using PySpark and Databricks.
• Implement data governance and compliance practices within enterprise data ecosystems.
• Identify and resolve data quality issues proactively through automated validation and monitoring.
• Track and document data lineage across multiple data sources and transformation layers.
• Write clean, scalable, and maintainable Python code for data engineering workflows.
• Collaborate effectively with cross‑functional teams including data scientists, analysts, and platform engineers.
• Troubleshoot and optimize data pipeline performance, reliability, and cost efficiency in Azure environments.

What We Bring to the Table

• Opportunity to work on large‑scale cloud data platforms and advanced data engineering solutions.
• Exposure to modern data stack technologies including Databricks, Azure Data Factory, and distributed processing frameworks.
• Collaborative environment working with data scientists, analytics teams, and engineering professionals.
• Challenging projects focused on building scalable and enterprise‑grade data platforms.
• Opportunities to enhance expertise in data governance, data quality, and lineage frameworks.

Let’s Connect

Want to discuss this opportunity in more detail? Feel free to reach out.