Senior Fabric Data Engineer (ID#5544)

New Value Solutions Group; Vancouver; On-site; CAD 96,000 - 152,000; Full time

Job summary

New Value Solutions is seeking a Senior Microsoft Fabric Data Engineer who will hands-on build Microsoft Fabric environments, CI/CD pipelines, a metadata-driven onboarding framework and pilot source onboarding for client projects.

The role is a contract engagement (300-400 hours over 3-4 months) with potential for additional work on other client projects. You will collaborate with client engineers, implement robust data pipelines, and ensure smooth onboarding and operations.

Qualifications

• Two+ years building on Microsoft Fabric in production: Lakehouse, notebooks (PySpark and Spark SQL), Data Factory pipelines, Copy jobs, Dataflows Gen2, Mirroring, Delta tables and table maintenance.
• Azure DevOps YAML pipelines and Git workflows; fabric-CICD deployment via REST APIs.
• Infrastructure as code with Bicep or Terraform for Azure networking and platform resources.
• Experience with VNet data gateways and private endpoints.

Responsibilities

• Build Dev, Test and Prod environments per foundation design using Bicep and Microsoft Fabric REST APIs for capacity, domains and workspaces.
• Verify private endpoint access, gateway connectivity, and mirroring on data.
• Implement source control and CI/CD: Git, Azure DevOps pipelines, fabric-CICD deployments in client VNets; manage parameters and releases.
• Create metadata-driven onboarding framework: source registry, templates, validation and reconciliation.
• Onboard pilot sources into Bronze/Silver/Gold; schedule and monitor loads; run reconciliation, freshness and recovery tests.
• Write operations and onboarding playbooks and environment verification checklists.
• Partner with client engineers; deliver knowledge transfer workshops on CI/CD and onboarding.

Skills

Microsoft Fabric

PySpark

Spark SQL

Data Factory pipelines

Copy jobs

Dataflows Gen2

Delta tables

Mirroring

Lakehouse

Table maintenance

Azure DevOps

Git

Bicep

Terraform

SQL

Python

Fabric REST APIs

Tools

Azure DevOps

Git

Bicep

Terraform

Fabric REST APIs

Job description

New Value Solutions is a national IT consulting company. We are searching a Senior Microsoft Fabric Data Engineer who is a hands on builder of Microsoft Fabric environments, the CI/CD pipeline, the metadata driven onboarding framework and the pilot source onboarding.

This is a contract role, estimated to be 300-400 hours over a 3-4 month period. Definite possibilities for more hours as needed. We also have another project for a different client which may require additional assistance.

Responsibilities

• Build Dev, Test and Prod exactly as approved in the foundation design, using Bicep for Azure resources (capacity, gateway subnet, private endpoints, private DNS, Key Vault) and the Microsoft Fabric REST APIs for capacity assignment, domains, workspaces, private links, managed private endpoints, gateway connections and Entra ID group role assignments.
• Run the connectivity proof on non sensitive data to confirm private endpoint access, gateway connectivity and mirroring behaviour under the proposed network model.
• Implement source control and CI/CD: Git integration for workspaces, Azure DevOps Repos and Pipelines, fabric-CI/CD deployment from a build agent inside the client's VNet, parameter files and variable libraries, approval gates and release notes. Demonstrate a change moving Dev to Test to Prod.
• Build the metadata driven onboarding framework: source registry, pipeline and notebook templates, validation and reconciliation templates.
• Onboard the approved pilot source into Bronze, Silver and Gold, schedule and monitor loads, and run reconciliation, freshness, performance, access and recovery tests.
• Write the operations runbook, the onboarding playbook and the environment verification checklist.
• Pair with client engineers throughout and coach them while they onboard part of the pilot themselves.
• Deliver knowledge transfer workshops on CI/CD and release process, onboarding a new source and operations and troubleshooting.

Required experience and skills

• At least two years building on Microsoft Fabric in production: Lakehouse, notebooks (PySpark and Spark SQL), Data Factory pipelines, Copy jobs, Dataflows Gen2, Mirroring, Delta tables and table maintenance.
• Azure DevOps YAML pipelines and Git workflows, including branch policies and pull request review. Experience deploying Fabric items with fabric-CICD or the Fabric REST APIs.
• Infrastructure as code with Bicep (or Terraform) for Azure networking and platform resources.
• Practical experience with VNet data gateways or on premises data gateways and with sources behind private endpoints.
• Strong SQL and Python. Able to design metadata driven, configuration first frameworks rather than one off pipelines.
• Data validation and reconciliation practices: row counts, checksums, freshness checks, failure handling and alerting.
• Able to explain what they build to client engineers and to document it clearly.

Preferred Skills, an asset

• DP-700 (Fabric Data Engineer) or DP-600 certification.
• Oracle and Azure SQL Database source experience, including LogMiner or change data capture prerequisites.
• Power BI semantic model basics, Microsoft Purview, Fabric capacity monitoring and cost management.
• Experience in public sector environments with formal security review.