Azure Databricks Engineer

Robotic Process Automation LLC; Wilmington (DE); On-site; USD 120,000 - 160,000; Full time

Job summary

Robotic Technologies Automation in Wilmington, Delaware, is seeking a senior data engineering lead to drive the end-to-end migration from Informatica to Azure Databricks. You will design target state architecture, define migration strategy, and ensure a smooth transition from batch ETL on Oracle to modern ELT pipelines.

You will translate complex mappings to PySpark, lead technical decisions, collaborate with security and operations, establish success criteria, and mentor teams to achieve

Responsibilities

• Lead end-to-end migration of data pipelines, workflows, and integration patterns from Informatica to Azure Databricks.
• Assess current Informatica assets and define target architecture, migration strategy, and phased execution plans.
• Modernize ETL patterns into scalable ELT using Azure Databricks best practices.
• Translate Informatica PowerCenter mappings into PySpark pipelines in Databricks.
• Evaluate Oracle stored procedures and migrate/refactor logic in Databricks with PySpark or Spark SQL.
• Provide technical leadership on design, standards, testing, deployment, and production readiness.
• Collaborate with business, product, architecture, security, and operations to align migration with enterprise priorities.
• Establish migration success criteria for data quality, reconciliation, performance, resilience, and observability.
• Identify risks and dependencies early; drive mitigation plans.
• Guide cutover, stabilization, and post-migration optimization for platform adoption.
• Mentor engineers on Databricks design patterns and modern data engineering practices.

Job description

• Lead the end to end migration of data pipelines, workflows, and integration patterns from Informatica to Azure Databricks.
• Assess current state Informatica assets and define target state architecture, migration strategy, and phased execution plans. The current environment is based on Informatica PowerCenter 10.4.1 running batch ETL workloads, with Oracle as both the source and target database. The target state platform is Azure Databricks (Runtime 14.4 LTS) with Unity Catalog, Delta Live Tables (DLT), and ADLS Gen2.
• Drive modernization of ETL patterns into scalable ELT and data engineering approaches aligned with Azure Databricks best practices.
• Analyze and interpret complex Informatica PowerCenter mappings, including transformations, lookups, joins, filters, routers, and expression logic, and build equivalent PySpark based pipelines in Azure Databricks.
• Assess Oracle database level stored procedures, functions, and packages that are invoked by or tightly coupled with Informatica workflows, and determine whether to migrate, refactor, or recreate that logic within Databricks using PySpark, Python, or Spark SQL.
• Provide technical leadership across solution design, engineering standards, code quality, testing strategy, deployment, and production readiness.
• Partner with business, product, architecture, infrastructure, security, and operations stakeholders to align migration outcomes with enterprise priorities.
• E stablish migration success criteria covering data quality, reconciliation, performance, resilience, observability, and business continuity.
• Identify delivery risks, dependencies, and technical constraints early, and drive mitigation plans through execution.
• Guide cutover, stabilization, and post migration optimization to ensure sustainable platform adoption and operational excellence.
• Mentor engineers and technical teams on Databricks design patterns, reusable frameworks, and modern data engineering practices

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.