Azure Databricks Lead: Lakehouse & Spark Architect

HCLTech; New York (NY); Hybrid; USD 127,000 - 180,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401(k) retirement plan

Paid time off

10 paid holidays per year

Job summary

HCLTech in New York is seeking an Azure Databricks Technical Lead to design, build, and optimize scalable data engineering solutions in a hybrid, enterprise environment. You will leverage Azure Databricks, ADF and ADLS Gen2 to deliver cloud-native data lakehouse architectures and enable data-driven decisions.

The role emphasizes strong Python expertise, PySpark, Spark SQL, and Delta Lake, plus collaboration with stakeholders and architects to drive performance and governance across large

Qualifications

• Bachelor's or Master's degree in CS, IT, engineering, data analytics or related field.
• Strong Python and PySpark knowledge.
• Experience with Delta Lake and cloud data platforms on Azure.
• Experience with data governance, security, and scalable data architectures.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks, ADF and ADLS Gen2.
• Build and optimize batch and real-time data pipelines with PySpark and Spark SQL.
• Develop Python applications following OOP and advanced design patterns.
• Implement Delta Lake-based data solutions to support lakehouse architectures.
• Collaborate with stakeholders and architects to deliver scalable cloud data solutions.
• Monitor and optimize data pipelines and ensure data quality and performance.

Skills

Python

PySpark

Spark SQL

SQL

OOP concepts

Design patterns

Education

Bachelor's degree in CS/IS/Engineering

Master's degree in related field

Tools

Azure Databricks

Azure Data Factory

ADLS Gen2

Delta Lake

Azure Synapse

Azure analytics services

Job description

HCLTech in New York is seeking an Azure Databricks Technical Lead to design, build, and optimize scalable data engineering solutions in a hybrid, enterprise environment. You will leverage Azure Databricks, ADF and ADLS Gen2 to deliver cloud-native data lakehouse architectures and enable data-driven decisions.

The role emphasizes strong Python expertise, PySpark, Spark SQL, and Delta Lake, plus collaboration with stakeholders and architects to drive performance and governance across large