Azure Databricks Engineer/Lead/ Architect

HCLTech; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

HCLTech in Bengaluru, India, seeks an Azure Databricks Data Engineer to design and optimize modern data platforms based on Lakehouse architecture. You will build robust ETL/ELT pipelines with PySpark, manage Delta Live Tables, and ensure governance and cost efficiency across data pipelines.

The role focuses on scalable data solutions, automated ingestion with Autoloader, and advanced SQL for analytics at scale, enabling data-driven decision making for complex workloads.

Qualifications

• Data Lakehouse with Medallion Architecture Bronze-Silver-Gold.
• Unity Catalog and data governance fundamentals.
• Advanced SQL proficiency.
• Data quality and validation frameworks.
• Pipeline performance tuning and optimization.

Responsibilities

• Design and implement Data Lakehouse architectures using the Medallion Framework.
• Develop high-performance data pipelines using PySpark notebooks and Databricks.
• Build and manage Delta Live Tables pipelines for reliable processing.
• Implement automated data ingestion using Autoloader for batch and streaming.
• Ingest and process data from diverse sources including files, databases, APIs, and streaming.
• Perform ETL/ELT transformations to support analytics and reporting.
• Ensure data governance, security, and access control using Unity Catalog.
• Monitor, optimize, and troubleshoot pipelines for performance and cost.
• Implement robust data quality checks, validations, and monitoring.
• Develop and optimize advanced SQL queries for large-scale data workloads.

Skills

Azure Databricks

Data Engineer

PySpark

Delta Live Tables

Autoloader

Unity Catalog

Data Governance

Data Lakehouse

Advanced SQL

Data Quality

ETL/ELT

Delta Lake

Job description

We are seeking a highly skilled Azure Databricks Data Engineer to build and optimize modern data platforms leveraging the Lakehouse architecture. If you are passionate about designing scalable data solutions, implementing robust ETL/ELT pipelines, and enabling data-driven insights through Databricks, we'd love to hear from you.

What We're Looking For

Key Responsibilities

• Design and implement Data Lakehouse architectures using the Medallion Framework (Bronze, Silver, Gold layers).
• Develop high-performance data pipelines using PySpark notebooks and Databricks.
• Build and manage Delta Live Tables (DLT) pipelines for reliable and efficient data processing.
• Implement automated data ingestion using Autoloader for batch and streaming workloads.
• Ingest and process data from diverse sources including files, databases, APIs, and streaming platforms.
• Perform complex ETL/ELT transformations to support analytics and reporting requirements.
• Ensure data governance, security, and access control using Unity Catalog.
• Monitor, optimize, and troubleshoot pipelines for performance, scalability, and cost efficiency.
• Implement robust data quality checks, validations, and monitoring frameworks.
• Develop and optimize advanced SQL queries for large-scale data workloads.

Mandatory Skills

Data Lakehouse / Medallion Architecture (Bronze-Silver-Gold)

Unity Catalog & Data Governance

Advanced SQL

Data Quality & Validation Frameworks

Pipeline Performance Tuning & Optimization

Keywords: Azure Databricks | Data Engineer | PySpark | Delta Live Tables | DLT | Autoloader | Unity Catalog | Data Governance | Data Lakehouse | Medallion Architecture | ETL | ELT | Advanced SQL | Data Quality | Delta Lake