Data Engineer

Provoke Solutions; Auckland; On-site; NZD 80,000 - 110,000; Full time

Job summary

A leading software solutions provider in Auckland is looking for an experienced Data Engineer to design and manage scalable data lakes using MS Fabric. The role involves developing and maintaining data pipelines, ensuring data quality, and working closely with cross-functional teams. The ideal candidate will have strong programming skills in Python and SQL, along with hands-on experience in cloud platforms like Azure. Join this dynamic team and leverage your expertise to drive exceptional outcomes.

Qualifications

• Proven experience as a Data Engineer, focused on data lake architecture.
• Hands-on experience with MS Fabric and building data lakes.
• Proficiency in data integration techniques for structured and unstructured data.

Responsibilities

• Design, implement, and manage data lake solutions using MS Fabric.
• Develop and maintain data pipelines for ETL processes.
• Collaborate with teams to understand data requirements.

Skills

Data lake architecture

MS Fabric

Data integration

Python

SQL

Azure

AWS

Power BI

Data governance

Problem-solving

Tools

Apache Spark

Hadoop

Databricks

Job description

Join the dynamic team at Provoke, where we're not just about meeting expectations but exceeding them. We're looking for innovative professionals who are passionate about driving exceptional outcomes for our customers. At Provoke, you'll be empowered to challenge the status quo and encouraged to think differently, leveraging a growth mindset to deliver tangible results. Here, professional growth isn't just a concept-it's a reality, fueled by continuous learning and a supportive, forward-thinking environment. If you're ready to make a significant impact and grow alongside a team of like-minded individuals, Provoke is your destination.

What we do

We build bespoke software using modern technologies and are on a mission to help our clients flourish with smart solutions to solve their business needs.

The Provoke Experience

We are committed to building high performing teams and offer tangible rewards to ensure that effort is recognized. We are dedicated to providing an enriching environment and learning opportunities for our employees to grow within and fast track their careers. We are focused on diversity diversity in race, gender, orientation and experience.

Job Overview

As a Data Engineer, you will be responsible for designing, developing, and managing scalable data pipelines that feed into our data lake using MS Fabric. You will work closely with our Lead Data Engineer, analysts, and other stakeholders to ensure that the data infrastructure supports both current and future needs. Your role will involve integrating various data sources, ensuring data quality, and building a robust, scalable data lake architecture.

Key Responsibilities

• Design, implement, and manage data lake solutions using MS Fabric.
• Develop and maintain data pipelines to extract, transform, and load (ETL) data from various structured and unstructured sources.
• Collaborate with cross-functional teams to understand data requirements and ensure seamless integration of new data sources into the data lake.
• Optimize data storage and retrieval processes to improve performance and scalability.
• Ensure the integrity, security, and availability of the data lake by implementing best practices in data governance and management.
• Perform data profiling, cleansing, and transformation to ensure data quality.
• Monitor and troubleshoot data flows to ensure reliable operation of the data pipelines.
• Stay up to date with the latest trends and technologies in data engineering, particularly in relation to MS Fabric and data lake architectures.

Requirements

• Proven experience as a Data Engineer, with a strong focus on data lake architecture and development.
• Hands-on experience with MS Fabric and building data lakes.
• Proficiency in data integration techniques for structured and unstructured data from multiple sources (e.g., APIs, databases, cloud services).
• Strong programming skills in Python, SQL, or other relevant languages.
• Experience with cloud platforms like Azure, AWS, or Google Cloud for data storage and processing.
• Solid understanding of data warehousing, ETL processes, and big data technologies.
• Strong experience with Power BI and DAX for data visualization
• Knowledge of data governance principles and practices.
• Strong problem-solving skills with attention to detail.
• Excellent communication and collaboration skills.

Preferred

• Experience with additional data tools and platforms, such as Apache Spark, Hadoop, or Databricks.
• Familiarity with machine learning workflows and model deployment in data lake
• Experience working in Agile or DevOps environments.