Data Engineer

InterEx Group; Noord-Holland; On-site; EUR 65,000 - 90,000; Full time

Job summary

InterEx Group seeks an experienced Data Engineer to design and implement data pipelines, ensuring scalable transformation and integration of large datasets for analytics and ML workloads.

You will work with Azure-based tools, collaborate with data governance and architects, and help elevate data quality and accessibility across the organization.

Qualifications

• Bachelor’s or Master’s Degree in Computer Science required.
• Minimum 4 years of experience as a data engineer.
• Experience with Azure SQL, Azure Data Factory, Azure DevOps, Databricks and Delta Lake is a must.
• Experience with high-volume heterogeneous data sets is an asset.
• Experience with agile software processes and data-driven development.
• Experience with data lakes and dimensional models.

Responsibilities

• Works with state-of-the-art data storage and processing technologies.
• Builds large-scale batch and real-time data pipelines in cloud platforms or on-premise.
• Pre-processes, cleanses and structures data for analytics and ML activities.
• Liaises with data governance and architects to align on requirements.
• Participates in cross-functional agile teams to deliver new product objectives.
• Implements automation to improve integration flows and data quality.

Skills

ETL/ELT development

Advanced data modeling

Problem solving

Independent work

Agile development

Data governance awareness

Education

Bachelor’s or Master’s Degree in Computer Science

Tools

Azure SQL

Azure Data Factory

Azure DevOps

Databricks

Delta Lake

Job description

The Data Engineer extracts, combines, transforms, loads, manipulates and cleanses data all whilst effectively handling interfaces to large amounts of data and creates/integrates data structures suitable for analysis.

The Data Engineer must understand data architecture and data management. Develop and deploys algorithms in performance-efficient code in analytical and data platforms. Liaise with enterprise architects, data governors, and subject matter experts to fill any gaps in the requirements or design.

Core Responsibilities:

• Works and experiments with state-of-the-art data storage and processing technologies.
• Builds large-scale batch and real-time data pipelines with data processing frameworks in cloud platforms or on-premise
• Leverages best practices in continuous integration and delivery
• Pre-process, clean and structures data to facilitate data exploration and advanced analytics/Machine Learning activities
• Learns and uses modern data preparation, integration and AI-enabled metadata management tools and techniques.
• Tracks data consumption patterns, working with data governance and information stewards within these teams, and participates in vetting and promoting content created in the business and by data scientists to the curated data catalogue for governed reuse
• Performs intelligent sampling and caching.
• Recommends and implements automation in existing and future integration flows.
• Helps improve data quality by driving advancements in data acquisition, testing and tooling
• Works in cross-functional agile teams to continuously experiment, iterate and deliver on new product objectives
• Analyzes requested changes to fill in the details fitting in the complete picture and while doing that aligns with all parties involved (Architects, Data Governance, …), either as preparation for yourself or for data engineer colleagues, before starting the implementation.

Education & Experience:

• Bachelor’s or Master’s Degree in Computer Science preferred or equivalent combination of education and experience
• Minimum 4 years of experience as a data engineer
• Experience in (captive) financial services and understands the processes and execution models in financial services is an asset
• Experience in Azure SQL, Azure Data Factory, Azure DevOps, Databricks, Delta Lake is a must, certification is recommended
• Experience in working with high-volume heterogeneous data sets 2
• Experience in agile software processes and data-driven development
• Experience with data lakes and dimensional models

Required Competencies and Skills:

• Azure Data Engineer Associate certification is recommended
• Databricks Data Engineer Professional certification is recommended
• Strive for clean, reusable and performant code
• Skilled ETL/ELT developer
• Strong understanding of data and information
• Business savvy, understand information from a business perspective and use that to more effectively develop enabling technologies
• Strong problem-solving and analytical skills
• Goal-oriented and positive attitude to deliver against targets
• Able to work independently and proactively
• Global, open, innovative, holistic and goal-oriented mindset
• Ability to set and meet deadlines
• Ability to work in high-pressure situations
• Excellent attention to detail
• Advanced level of English, oral and written