Azure Data Lead: Spark-on-Databricks Modernization Architect

Shrive Technologies; New York (NY); On-site; USD 170,000 - 250,000; Full time

Job summary

Shrive Technologies seeks an Azure Senior Data Lead to guide the modernization of Python applications into scalable PySpark solutions on Azure Databricks. You will blend software engineering, data engineering, cloud architecture, and performance optimization to deliver enterprise-grade data platforms.

The role requires leading design and implementation of Delta Lake architectures, data pipelines, and quality frameworks across Azure services, with emphasis on scalable, cost-efficient outcomes and

Qualifications

• Senior data engineering leader with proven delivery of large-scale Databricks modernization programs.
• Must convert Python applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks.

Responsibilities

• Analyze existing Python OOP applications and redesign single-node Processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using Bronze Silver Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs (partitioning, bucketing, caching, broadcast joins, AQE, Delta optimization) and benchmark against originals.

Skills

Python

OOP

advanced Python design patterns

PySpark

Spark SQL

SQL

Azure Databricks

Azure Data Factory

ADLS Gen2

Apache Spark

Delta Lake

Data Lakehouse architecture

distributed computing

Job description

Shrive Technologies seeks an Azure Senior Data Lead to guide the modernization of Python applications into scalable PySpark solutions on Azure Databricks. You will blend software engineering, data engineering, cloud architecture, and performance optimization to deliver enterprise-grade data platforms.

The role requires leading design and implementation of Delta Lake architectures, data pipelines, and quality frameworks across Azure services, with emphasis on scalable, cost-efficient outcomes and