Azure Databricks Migration Lead

MACHINE LEARNING TECHNOLOGIES LLC; Wilmington (DE); On-site; USD 140,000 - 170,000; Full time

Job summary

MACHINE LEARNING TECHNOLOGIES LLC is seeking an Azure Databricks Engineer in Wilmington, DE. The role focuses on migrating from Informatica to Azure Databricks, designing the target state, and delivering scalable ELT data pipelines using PySpark and Spark SQL in a Databricks environment.

The position requires technical leadership, collaboration with enterprise stakeholders, and ensuring data quality, performance, and observability through the migration lifecycle.

Responsibilities

• Lead the end to end migration of data pipelines, workflows, and integration patterns from Informatica to Azure Databricks.
• Assess current state Informatica assets and define target state architecture, migration strategy, and phased execution plans.
• Drive modernization of ETL patterns into scalable ELT and data engineering approaches aligned with Azure Databricks best practices.
• Analyze and interpret complex Informatica PowerCenter mappings and build equivalent PySpark based pipelines in Azure Databricks.
• Assess Oracle database level stored procedures and determine migration/refactor/recreation in Databricks using PySpark/Spark SQL.
• Provide technical leadership across solution design, engineering standards, testing strategy, deployment, and production readiness.
• Partner with stakeholders to align migration outcomes with enterprise priorities.
• Establish migration success criteria covering data quality, reconciliation, performance, resilience, observability, and business continuity.
• Identify delivery risks and drive mitigations; guide cutover, stabilization, and post-migration optimization.
• Mentor engineers on Databricks design patterns and modern data engineering practices

Job description

MACHINE LEARNING TECHNOLOGIES LLC is seeking an Azure Databricks Engineer in Wilmington, DE. The role focuses on migrating from Informatica to Azure Databricks, designing the target state, and delivering scalable ELT data pipelines using PySpark and Spark SQL in a Databricks environment.

The position requires technical leadership, collaboration with enterprise stakeholders, and ensuring data quality, performance, and observability through the migration lifecycle.