Senior Azure Data Migration & Dataverse Integration Engineer

Arch Systems, LLC; Baltimore (MD); Remote; USD 120,000 - 180,000; Full time

Job summary

Arch Systems is seeking a Senior Azure Data Migration & Dataverse Integration Engineer to own enterprise data migration and synchronization between SQL Server and Dataverse. The role covers source analysis, data mapping, extraction, transformation, migration, and automated deployment across DEV, TEST, UAT, and PROD environments.

The ideal candidate will lead end-to-end data pipelines, build automated ADF pipelines, and ensure data quality, integrity, and performance in complex Power Platform

Qualifications

• Advanced hands-on experience designing and developing Azure Data Factory solutions.
• Strong SQL Server data engineering and T-SQL development skills.
• Experience migrating enterprise SQL data into Dataverse and managing data quality.
• Knowledge of CDC, incremental loads, and data synchronization patterns.
• Experience implementing automated data validation, reconciliation, and monitoring.

Responsibilities

• Design and implement end-to-end SQL Server–Dataverse data migration pipelines.
• Develop automated pipelines using Azure Data Factory and supporting Azure services.
• Analyze legacy databases and map data structures to Dataverse models.
• Implement CDC, incremental loads, and ongoing synchronization.
• Develop logging, auditing, exception handling, and deployment automation.
• Ensure referential integrity and proper parent/child relationships during migration.
• Monitor migrations, troubleshoot failures, and optimize performance.

Skills

Azure Data Factory

SQL Server

Dataverse

Data Migration

CDC

Azure services

Data pipelines

T-SQL

DevOps

Tools

Dataverse API

Power Platform

REST API

Azure SQL

CI/CD pipelines

Job description

Company: Arch Systems

Location: Remote

Type: Full-time

Position Summary

Seeking a Senior Azure Data Migration & Dataverse Integration Engineer to design, develop, implement, and support enterprise data migration and synchronization capabilities between Microsoft SQL Server and Microsoft Dataverse.

The position requires a senior hands-on engineer capable of owning the complete data pipeline lifecycle, including source analysis, data mapping, extraction, transformation, migration, synchronization, validation, exception handling, monitoring, performance optimization, and automated deployment.

The engineer will develop and maintain repeatable pipelines supporting initial bulk migration, incremental migration, Change Data Capture (CDC), ongoing synchronization, and transition/cutover activities between legacy SQL Server applications and modern Microsoft Power Platform/Dataverse solutions.

Primary Responsibilities

The Data Migration Engineer will:

• Design and implement end-to-end SQL Server-to-Dataverse data migration pipelines.


• Develop automated pipelines using Azure Data Factory (ADF) and supporting Azure services.


• Analyze legacy SQL Server databases, tables, views, relationships, stored procedures, and data dependencies.


• Map legacy SQL Server data structures to Dataverse tables, columns, relationships, choices, lookups, and alternate keys.


• Change Data Capture


• Scheduled synchronization


• Dataverse-to-legacy synchronization where required


• Cutover and final synchronization


• Develop and maintain ADF pipelines, datasets, linked services, integration runtimes, triggers, parameters, variables, and reusable pipeline components.


• Develop transformation and data-cleansing logic required to convert legacy data into Dataverse-compatible structures.


• Implement controlled sequencing to maintain parent/child relationships and referential integrity.


• Develop lookup resolution and cross-reference processes for Dataverse GUIDs and legacy SQL identifiers.


• Implement restartable and recoverable migration processes.


• Develop pipeline logging, audit, exception handling, retry, and reconciliation capabilities.


• Validate migrated data for completeness, accuracy, integrity, and consistency.


• Perform source-to-target record-count and field-level reconciliation.


• Troubleshoot failed records, pipeline failures, API limitations, data-quality issues, and performance problems.


• Develop automated deployment processes for data pipeline components across DEV, TEST, UAT, and PROD environments.

The candidate must have advanced hands-on experience designing and developing Azure Data Factory solutions.

Responsibilities include:

• ADF pipeline architecture and development


• Copy Activities


• Lookup Activities


• Stored Procedure Activities


• ForEach and Until processing


• Conditional processing


• Dynamic expressions


• Linked Services


• Datasets


• Integration Runtime configuration


• Managed Identity authentication


• Pipeline triggers and scheduling


• Incremental load processing


• CDC patterns


• Retry and failure handling


• Pipeline logging and auditing


• Performance tuning and parallelization


• Environment-specific configuration


• Infrastructure and pipeline deployment automation

The engineer should be capable of developing reusable, metadata-driven pipeline patterns rather than creating independent hard‑coded pipelines for every table.

SQL Server Responsibilities

Strong SQL Server development and data engineering skills are required, including:

• Complex T-SQL development


• Stored procedures


• Views


• Common Table Expressions


• Temporary and staging tables


• Data profiling


• Data cleansing


• Data transformation


• Joins and relationship analysis


• Primary and foreign key analysis


• Indexing and query optimization


• SQL Server Change Data Capture


• Change Tracking


• Watermark-based incremental extraction


• Referential-integrity validation

The engineer must be capable of analyzing legacy databases with limited documentation and determining the technical relationships necessary to migrate data successfully.

Microsoft Dataverse Responsibilities

Advanced Dataverse knowledge is required, including:

• Dataverse tables and columns


• Standard and custom tables


• Table relationships


• One-to-many and many-to-many relationships


• Lookups


• Choice columns


• Alternate keys


• GUID management


• Ownership and security considerations


• Dataverse Web API


• Dataverse APIs and connectors


• Batch processing


• Dataverse service protection/API limits


• Application users and service principals


• Solutions and ALM

The candidate must understand how Dataverse differs from a traditional relational SQL database and design migration processes accordingly.

The engineer will design synchronization processes supporting coexistence between legacy and modernized applications during phased transition.

This includes:

• Dataverse to SQL Server synchronization where required


• Unidirectional and bidirectional synchronization patterns


• CDC and watermark processing


• Event-driven integration where applicable


• Conflict detection and resolution


• Duplicate prevention


• Failed-record reprocessing


• Reconciliation between source and target systems

The engineer must understand the risks associated with bidirectional synchronization, including update conflicts, synchronization loops, transaction ordering, and determination of the authoritative system of record.

The candidate should be capable of implementing a controlled migration architecture such as:

The architecture must support traceability from the original source record through transformation and final Dataverse record creation.

Pipeline Monitoring and Operations

Develop operational capabilities to monitor migration and synchronization processing, including:

• Records transformed


• Records successfully loaded


• Warning and error counts


• Source and target record counts


• Retry status

Implement monitoring using applicable Microsoft technologies such as:

• Application Insights


• ADF monitoring


• Power BI or operational dashboards

DevOps and CI/CD

The engineer will establish controlled deployment and configuration management for all data migration components.

Responsibilities include:

• Branching and source-control practices


• Pull requests and code reviews


• Automated DEV/TEST/UAT/PROD deployments


• Environment-specific parameters


• Infrastructure as Code


• Bicep and/or ARM templates


• Dataverse configuration coordination


• Key Vault and secret-management integration


• Release validation


• Rollback procedures

All pipelines, scripts, SQL objects, infrastructure definitions, configuration, and supporting code must be maintained under source control.

Security Requirements

The engineer will implement data migration capabilities consistent with enterprise security requirements, including:

• Managed Identities


• Dataverse Application Users


• Role-Based Access Control


• Secure connection management


• Encryption in transit and at rest


• Audit logging


• Environment separation


• Controlled production access

Experience supporting Federal Government, DoD, RMF, NIST 800-53, or comparable regulated environments is preferred.

Required Technical Skills

• 7+ years of data engineering, database development, integration, or data migration experience.


• 4+ years of Microsoft Azure data engineering experience.


• Advanced Azure Data Factory development experience.


• Advanced Microsoft SQL Server and T-SQL experience.


• Experience migrating enterprise SQL data into Dataverse.


• Experience with CDC, incremental loads, delta processing, and synchronization.


• Experience with REST APIs and Dataverse Web API.


• Experience with Azure SQL.


• Experience with Azure Key Vault and Managed Identities.


• Experience with Azure DevOps source control and pipelines.


• Experience implementing automated data validation and reconciliation.

Preferred Technical Experience

Experience with several of the following is highly desirable:

• Microsoft Fabric


• Logic Apps
  
• Power Automate


• Power Apps


• Power BI


• Synapse Analytics


• C#/.NET


• Python


• PowerShell


• Bicep


• Git

Preferred Certifications

One or more of the following Microsoft certifications is preferred:

• DP-203 – Azure Data Engineer Associate or current Microsoft equivalent


• Microsoft Azure Administrator, Developer, or Solutions Architect certification

Desired Candidate Profile

This position requires a hands-on senior data engineer, not solely a data architect or Power Platform administrator.

The successful candidate must be able to independently take a legacy SQL data requirement through the complete technical lifecycle:

The engineer must be capable of working directly with application developers, Power Platform developers, database administrators, solution architects, business analysts, and system owners to determine how legacy data will be migrated, synchronized, validated, and ultimately transitioned from the legacy environment.

The position will be responsible for producing and maintaining:

• Data synchronization pipelines


• SQL extraction and transformation scripts


• CDC/incremental processing framework


• Data-quality rules


• Exception and retry processes


• Cutover procedures


• Technical architecture and data-flow diagrams