Lead Data Consultant (Microsoft Fabric)

Trility Consulting; Virginia; À distance; USD 130 000 - 180 000; Plein temps

Avantages offerts par ce poste

Paid time-off

Company holidays

Résumé du poste

Trility Consulting is seeking an experienced Senior Data Consultant to modernize an Azure Synapse data platform and migrate workloads to Microsoft Fabric. You will partner with client engineers to design scalable ingestion frameworks and build data pipelines in a lakehouse environment.

You will implement CI/CD for Fabric artifacts, leverage Claude Code for AI-assisted development, and validate data migrations with robust testing and governance considerations.

Qualifications

• 5+ years of experience with Azure Data Services including Fabric, ADF, Databricks or Synapse.
• Hands-on Fabric experience in production environments with lakehouses and pipelines.
• Proven ETL/ELT pipeline craftsmanship and reusable ingestion frameworks.
• Ability to design a metadata-driven ingestion framework with config-based onboarding.
• Spark notebooks and data transformation using PySpark and Spark SQL.
• Strong SQL skills including T-SQL for data transformation and validation.
• Experience with Azure Synapse and understanding complex data dependencies.
• Experience building Git-based CI/CD workflows for data artifacts with GitHub Actions/Azure DevOps.
• Hands-on use of Claude Code for AI-assisted development and code review.
• Experience validating data migrations, ensuring quality, accuracy, and consistency.
• Data integration from REST APIs, relational databases, and structured/semi-structured data.
• Familiarity with governance, lineage, metadata management, and catalog concepts.
• Experience migrating data platforms from Azure Synapse to Fabric.
• Experience with Oracle and DB2 sources.
• Incremental ingestion, CDC, and source-specific loading strategies.
• Data reconciliation and validation frameworks for large migrations.
• Evaluating Purview or similar data catalog solutions.

Responsabilités

• Evaluate and refine the Fabric architecture, including metadata structures and reusable ingestion patterns.
• Design a metadata-driven ingestion framework allowing onboarding via configuration.
• Migrate prioritized sources from Azure Synapse to Microsoft Fabric.
• Develop and orchestrate Fabric data pipelines and Spark notebooks (PySpark, Spark SQL).
• Analyze Synapse pipelines to translate logic into Fabric solutions.
• Build data validation and reconciliation processes for migration accuracy.
• Develop loading strategies: full-load, incremental, or CDC per source.
• Implement Git-based CI/CD and environment promotion for Fabric artifacts.
• Leverage Claude Code to define specs, accelerate development, and review AI-generated code.
• Assess data dictionary, metadata, and lineage needs; explore Purview.
• Apply best practices for performance, testing, monitoring, security, and operations.

Connaissances

Azure Data Services

Microsoft Fabric

Azure Synapse

Azure Data Factory

Databricks

ETL/ELT pipelines

Spark notebooks

PySpark

Spark SQL

SQL

Data migration

CI/CD for data platforms

GitHub Actions

Azure DevOps

Claude Code

Data governance & catalog

CDC / incremental load

REST APIs integration

Oracle / DB2 experience

Outils

Claude Code

GitHub Actions

Azure DevOps

Microsoft Purview

Spark notebooks

Description du poste

Who We Are

At Trility Consulting®, we’ve built a strong, remote-work culture where people thrive, grow, and support each other – earning us Great Place to Work® Certification. Headquartered in Central Iowa, we embrace flexible schedules and collaboration across geographically distributed teams serving clients from all corners of the United States and globally.

Candidates must respect and value people, recognize that over-communication is barely good enough, thrive on solving complex problems, have a passion for building teams, and know that delivering what a client actually values is more important than their own predispositions. Candidates are self-motivated and relentlessly working to become more today than they were yesterday.

Trility Consulting is seeking an experienced Senior Data Consultant to support a client engagement focused on modernizing an existing Azure Synapse data platform and migrating prioritized workloads to Microsoft Fabric.

In this role, you’ll partner with Trility consultants and client engineers to pressure-test and refine the Fabric architecture, build reusable, metadata-driven ingestion frameworks, and migrate data workloads into a modern lakehouse environment. You’ll develop scalable data pipelines, implement validation processes to ensure migration accuracy, and establish automated CI/CD workflows that support reliable deployment from development through production.

You’ll work hands‑on alongside client engineers, contribute to architectural decisions, and help establish repeatable patterns that enable the team to maintain and extend the platform independently. Experience using Claude Code as part of an AI‑assisted development workflow is essential to this engagement.

This is a remote W2 position.

Responsibilities

• Evaluate and refine the existing Microsoft Fabric architecture, including metadata structures, table-level event triggering, and reusable ingestion patterns
• Design and implement a metadata‑driven ingestion framework that allows new tables and sources to be onboarded through configuration rather than custom development
• Migrate prioritized data sources and workloads from Azure Synapse into Microsoft Fabric
• Develop and orchestrate Fabric data pipelines and Spark notebooks using PySpark and Spark SQL
• Analyze existing Azure Synapse pipelines, including complex and nested workflows, to understand dependencies and translate existing processing logic into Fabric solutions
• Build data validation and reconciliation processes to verify migration accuracy, including duplicate detection, key alignment, and record‑level comparisons between Synapse and Fabric
• Analyze source‑specific loading requirements and recommend appropriate full‑load, incremental, or change data capture (CDC) patterns
• Implement Git‑based CI/CD and automated environment promotion for Fabric artifacts, including pipelines and notebooks, using GitHub Actions, Azure DevOps, or similar tooling
• Leverage Claude Code to define implementation specifications, accelerate development, and review AI‑generated code for quality, correctness, and maintainability
• Evaluate data dictionary, metadata, and lineage requirements, including potential catalog solutions such as Microsoft Purview
• Apply data engineering best practices related to performance, testing, monitoring, documentation, security, and operational support

Qualifications

• 5+ years of experience working with Azure Data Services, including Microsoft Fabric, Azure Data Factory, Databricks, or Synapse
• ~ Hands‑on Microsoft Fabric experience in production environments, including lakehouses, pipelines, notebooks, medallion architecture, workspaces, and capacity management
• ~ Strong experience building and maintaining production ETL/ELT pipelines and reusable data ingestion frameworks
• ~ Demonstrated experience designing and implementing a metadata‑driven ingestion framework, including configuration‑based onboarding and extensible processing patterns
• ~ Experience developing Spark notebooks and data transformation workflows using PySpark and Spark SQL
• ~ Strong SQL skills for data transformation, validation, and performance tuning, including the ability to read and understand existing T‑SQL logic
• ~ Experience working with Azure Synapse, including analyzing existing pipelines and understanding complex data processing dependencies
• ~ Experience designing and implementing Git‑based CI/CD workflows for data platform artifacts using GitHub Actions, Azure DevOps, or similar tooling
• ~ Hands‑on experience using Claude Code for AI‑assisted software development, including directing code generation, working across multiple files, and critically reviewing generated code
• ~ Experience validating data migrations and ensuring data quality, accuracy, and consistency across source and target systems
• ~ Experience integrating data from REST APIs, relational databases, and structured or semi‑structured data sources
• ~ Familiarity with data governance, lineage, metadata management, and data catalog concepts
• ~ Experience migrating data platforms or workloads from Azure Synapse to Microsoft Fabric
• Experience working with Oracle and DB2 data sources
• Experience implementing incremental ingestion, change data capture (CDC), and source‑specific loading strategies
• Experience developing data reconciliation and validation frameworks for large‑scale migrations
• Experience evaluating Microsoft Purview or other data catalog and lineage solutions

Location

Work remotely anywhere in the United States with flexible work hours

Benefits

• Paid time‑off, company holidays, and floating holidays