Azure Databricks Lead

HCLTech; New York (NY); Hybrid; USD 127,000 - 180,000; Full time

Benefits offered by this job

Medical Insurance

Dental Insurance

Vision Insurance

401(k) retirement plan

Paid time off

10 paid holidays per year

Job summary

HCLTech in New York is seeking an Azure Databricks Technical Lead to design, build, and optimize scalable data engineering solutions in a hybrid, enterprise environment. You will leverage Azure Databricks, ADF and ADLS Gen2 to deliver cloud-native data lakehouse architectures and enable data-driven decisions.

The role emphasizes strong Python expertise, PySpark, Spark SQL, and Delta Lake, plus collaboration with stakeholders and architects to drive performance and governance across large

Qualifications

• Bachelor's or Master's degree in CS, IT, engineering, data analytics or related field.
• Strong Python and PySpark knowledge.
• Experience with Delta Lake and cloud data platforms on Azure.
• Experience with data governance, security, and scalable data architectures.

Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks, ADF and ADLS Gen2.
• Build and optimize batch and real-time data pipelines with PySpark and Spark SQL.
• Develop Python applications following OOP and advanced design patterns.
• Implement Delta Lake-based data solutions to support lakehouse architectures.
• Collaborate with stakeholders and architects to deliver scalable cloud data solutions.
• Monitor and optimize data pipelines and ensure data quality and performance.

Skills

Python

PySpark

Spark SQL

SQL

OOP concepts

Design patterns

Education

Bachelor's degree in CS/IS/Engineering

Master's degree in related field

Tools

Azure Databricks

Azure Data Factory

ADLS Gen2

Delta Lake

Azure Synapse

Azure analytics services

Job description

HCLTech is looking for a highly talented and self-motivated Azure Databricks Technical Lead to join it in advancing the technological world through innovation and creativity.

Job ID: 156897

Position Type: Full-time (Hybrid)

Location: New York, NY

Role/Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS Gen2).
• Build and optimize batch and real-time data processing pipelines using PySpark and Spark SQL.
• Develop reusable, maintainable, and high-performance Python applications following OOP principles and advanced design patterns.
• Implement Delta Lake-based data solutions to support enterprise data lakehouse architectures.
• Design and optimize distributed data processing frameworks for large-scale datasets.
• Develop and maintain ETL/ELT processes ensuring data quality, reliability, and performance.
• Collaborate with business stakeholders, architects, and cross-functional teams to gather requirements and deliver scalable cloud data solutions.
• Monitor, troubleshoot, and optimize data pipelines and workloads running on Azure platforms.
• Support data governance, security, and compliance initiatives across enterprise data environments.
• Contribute to architecture discussions, code reviews, and best-practice implementation.
• Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• Strong hands-on experience with:
• Expert-level proficiency in Python programming.
• Strong understanding of Object-Oriented Programming (OOP) concepts and advanced Python design patterns.
• Extensive experience with:
• PySpark
• Spark SQL
• SQL
• Strong knowledge of:
• Data Lakehouse Architecture
• Distributed Computing Concepts
• Experience building scalable cloud-native data processing solutions.
• Strong analytical, troubleshooting, and performance optimization skills.
• Excellent communication and collaboration skills.

Level of Education

• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, Data Analytics, or a related technical discipline.
• Equivalent professional experience may be considered.

Desired Qualifications

• Experience with CI/CD pipelines and DevOps practices for data engineering.
• Exposure to Azure Synapse Analytics, Azure Fabric, or other Azure analytics services.
• Experience implementing enterprise-scale data lakehouse solutions.
• Knowledge of data governance, data quality, and security frameworks.
• Experience working in Agile/Scrum environments.

Pay and Benefits

Pay Range Minimum: $127,000 per year

Pay Range Maximum: $180,000 per year

HCLTech is an equal opportunity employer, committed to providing equal employment opportunities to all applicants and employees regardless of race, religion, sex, color, age, national origin, pregnancy, sexual orientation, physical disability or genetic information, military or veteran status, or any other protected classification, in accordance with federal, state, and/or local law. Should any applicant have concerns about discrimination in the hiring process, they should provide a detailed report of those concerns to [e-mail] for investigation.

A candidate's pay within the range will depend on their skills, experience, education, and other factors permitted by law. This role may also be eligible for performance-based bonuses subject to company policies. In addition, this role is eligible for the following benefits subject to company policies: medical, dental, vision, pharmacy, life, accidental death & dismemberment, and disability insurance; employee assistance program; 401(k) retirement plan; 10 days of paid time off per year (some positions are eligible for need-based leave with no designated number of leave days per year); and 10 paid holidays per year.

How You'll Grow

At HCLTech, we offer continuous opportunities for you to find your spark and grow with us. We want you to be happy and satisfied with your role and to really learn what type of work sparks your brilliance the best. Throughout your time with us, we offer transparent communication with senior-level employees, learning and career development programs at every level, and opportunities to experiment in different roles or even pivot industries. We believe that you should be in control of your career with unlimited opportunities to find the role that fits you best.