Senior Data Engineer (MS Fabric)

TechWize; New York, Northern (NY, KY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

TechWize seeks a Senior Data Engineer to build scalable data platforms on Microsoft Fabric in New York. You will ingest data from AWS, Azure, and direct pipelines, partnering with Solutions and technology teams in an agile setting.

Lead metadata-driven ingestion, enable AI analytics for student data, and ensure governance with Purview, while promoting DevOps practices for rapid, cost-efficient deployment.

Qualifications

• Experience building Microsoft Fabric-based data platforms.
• Ingest data from AWS, Azure, and direct pipelines.
• Knowledge of Bronze/Silver/Gold data layers and scalable modeling.
• Experience with CI/CD and version control practices.
• Experience deploying and managing Microsoft Purview for data governance.
• Familiarity with AI-driven analytics for student data.
• Collaborative work in Agile teams aligning business and DevOps.

Responsibilities

• Implement Metadata Driven Pipeline & DevOps Accelerator to streamline data platform development.
• Create data models for advanced analytics and AI use cases for student data.
• Ingest data from AWS, Azure, or direct pipelines into Microsoft Fabric.
• Set up layered data architecture (Bronze/Silver/Gold) aligned to data domains.
• Apply DevOps practices for scalable, repeatable data solutions.
• Deploy and manage Microsoft Purview for data lineage, cataloging, and governance.

Skills

Microsoft Fabric

Data ingestion

Data modeling

DevOps CI/CD

Purview

AI analytics

Agile collaboration

Data governance

Tools

CI/CD pipelines

Version control

Microsoft Purview

Job description

• 501, Fifth Avenue, Suite 805 New York, NY 10017

Position Title: Senior Data Engineer (MS Fabric)

Post Description: We are looking for experienced Senior Data Engineer to join our team to work on an exciting data platform project leveraging Microsoft Fabric. This role will involve collaborating closely with our Solutions Team and the University’s Technology teams in an agile environment. The ideal candidates will be responsible for building scalable data platforms, onboarding data from multiple source systems, and driving end-to-end data solutions. As a Senior Data Engineer, you will be key to implementing Metadata Driven ingestion framework along with DevOps Accelerator, ensuring rapid, cost-effective deployment.

Key Skill Set Requirements

• Expertise in building and managing Microsoft Fabric-based data platforms.


• Hands-on experience with data ingestion from multiple source systems (AWS, Azure, direct pipelines).


• Strong knowledge of data transformation and modeling (Bronze, Silver, Gold layers) with a focus on performance and scalability.


• Proven ability to work with DevOps tools and practices (CI/CD pipelines, version control, etc.).


• Experience deploying and managing Microsoft Purview or other data lineage and cataloging tools.


• Familiarity with AI-driven analytics solutions, particularly for detecting patterns and behaviors in student data.


• Strong collaboration skills, working in agile teams to align with both business and DevOps practices.

Responsibilities

• Implement Metadata Driven Pipeline & DevOps Accelerator to streamline the data platform implementation, using pre-developed code and best practices.


• Create a data model for enabling advanced analytics and AI use-cases to detect at-risk students and visualize student behaviour.


• Ingest data from existing and new source platforms, such as AWS, Azure, or direct pipelines, into Microsoft Fabric.


• Setup Layered Data Architecture and Transform and model data from Bronze, Silver, and Gold layers, ensuring alignment with CAUDIT Data Reference Model and defined data domains.


• Apply DevOps practices and tools for consistent, scalable, and repeatable data platform solutions.


• Deploy and manage Microsoft Purview for data lineage, cataloging, and governance, ensuring adherence to data governance controls.

Shift Time

Australian Shift