Azure Data Lead

Inherent Technologies; New York (NY); Hybrid; USD 140,000 - 170,000; Full time

Job summary

Inherent Technologies in NYC, NY is seeking an Azure Data Lead to spearhead modernization of Python applications into scalable PySpark solutions on Azure Databricks. The role requires deep expertise in distributed data processing and cloud architecture to deliver enterprise-grade data pipelines.

You will design and deploy Delta Lake architectures, build end-to-end ETL/ELT on Azure Data Factory and ADLS Gen2, and optimize Spark jobs for performance parity with Python implementations.

Qualifications

• Must-have includes Python, Azure Databricks, Azure Data Factory (ADF), MS SQL, Oracle PL/SQL.
• Senior data engineering leadership with proven delivery of large-scale Databricks modernization.
• Experience migrating Python OOP applications to PySpark on Azure Databricks.
• Certifications in Azure Data Factory, Azure Databricks, SQL, Oracle or Python are a plus.
• Strong software engineering, data engineering, cloud architecture and performance optimization skills.

Responsibilities

• Analyze existing Python OOP apps and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze Silver Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs (partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, Delta optimization) and benchmark converted applications against original Python implementations.

Skills

Python

PySpark

Spark SQL

Azure Databricks

Azure Data Factory

ADLS Gen2

Delta Lake

Data Lakehouse

Distributed computing

OOP

SQL

Performance optimization

Cloud architecture

Data pipelines

Tools

MS SQL

Oracle PL/SQL

Azure Data Factory

ADLS Gen2

Azure Synapse Analytics

Databricks

Spark

Job description

Position: Azure Data Lead

Location: NYC, NY *3 days onsite *

Duration: 1 Year

Job Description:

• Azure Data lead - Python, Pyspark, Databricks, ADF, Data Lake,
• Azure Senior Data Lead Leads modernization of Python applications into scalable PySpark solutions on Azure Databricks
• Lead the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks - bringing a strong blend of software engineering, data engineering, cloud architecture, and performance optimization.

Key Responsibilities

• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze Silver Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs (partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, Delta optimization) and benchmark converted applications against original Python implementations. Core Skills
• Python (expert), OOP, and advanced Python design patterns
• PySpark, Spark SQL, and SQL
• Azure Databricks, Azure Data Factory, ADLS Gen2
• Apache Spark, Delta Lake, Data Lakehouse architecture, distributed computing

Must-have (per requisition):

• Python, Azure Databricks, Azure Data Factory (ADF), MS SQL, Oracle PL/SQL.

Good to have:

• PySpark; certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python.
• Experience & Expected Outcome Senior data engineering leader with proven delivery of large-scale Databricks modernization programs.
• Expected outcome: existing Python applications converted into scalable, cost-efficient, enterprise-grade data solutions on Azure Databricks with proven performance parity.