Databricks Engineer - Senior/ Lead

SmartRecruiters, Inc.; Hyderabad; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Blend is seeking an experienced Azure Databricks Engineer to design, build, and optimize scalable data pipelines on the Azure cloud. You will work with Python, SQL, and Spark to enable analytics and ML use cases.

The role involves ETL/ELT workflows, Spark job tuning, and managing Delta Lake tables with ACID and time travel, in collaboration with data scientists and analysts. Security and governance in Azure are essential.

Qualifications

• 6+ years of experience in Data Engineering
• Strong hands-on experience with Azure Databricks
• Proficiency in Python for data processing
• Strong knowledge of SQL (joins, window functions, performance tuning)
• Hands-on experience with Spark / PySpark
• Experience with Delta Lake
• Knowledge of Azure Data Lake Storage (ADLS Gen2)
• Understanding of distributed computing concepts
• Experience with Git version control
• Exposure to cloud security and governance in Azure

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks
• Implement ETL/ELT workflows using PySpark, Spark SQL, and Python
• Optimize Spark jobs for performance, cost, and scalability
• Work with structured and semi-structured data (Parquet, Delta, JSON, CSV)
• Build and manage Delta Lake tables (ACID, time travel, schema evolution)
• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2)
• Develop complex queries and transformations using SQL
• Collaborate with data scientists, analysts, and stakeholders to support analytics and ML use cases
• Ensure data quality, validation, and monitoring
• Follow best practices for security, access control, and governance in Azure

Skills

Data engineering

Python

SQL

Spark

Tools

Azure Databricks

Apache Spark

PySpark

Delta Lake

Azure Data Lake Storage

Azure Data Factory

Git

RBAC in Azure

Job description

Job Description

Blend is a premier AI services provider, committed to co-creating meaningful impact for its clients through the power of data science, AI, technology, and people. With a mission to fuel bold visions, Blend tackles significant challenges by seamlessly aligning human expertise with artificial intelligence. The company is dedicated to unlocking value and fostering innovation for its clients by harnessing world-class people and data-driven strategy. We believe that the power of people and AI can have a meaningful impact on your world, creating more fulfilling work and projects for our people and clients. For more information, visit www.blend360.com .

We are looking for an experiencedAzure Databricks Engineerwith strong hands-onexpertiseinPython, SQL, and Apache Sparkto design, build, andoptimizescalable data pipelines and analytics solutions on theAzure cloud platform. The ideal candidate should have experience working with large datasets, distributed data processing, and modern data engineering practices.

Responsibilities

Design, develop, andmaintainscalabledata pipelines using Azure Databricks

ImplementETL/ELT workflowsusingPySpark, Spark SQL, and Python

OptimizeSpark jobs forperformance, cost, and scalability

Work with structured and semi-structured data (Parquet, Delta, JSON, CSV)

Build and manageDelta Laketables (ACID, time travel, schema evolution)

Integrate Databricks withAzure Data Lake Storage (ADLS Gen2)

Develop complex queries and transformations usingSQL

Collaborate with data scientists, analysts, and stakeholders to support analytics and ML use cases

Ensure data quality, validation, and monitoring

Follow best practices for security, access control, and governance in Azure

Qualifications

6+ years of experience inData Engineering

Strong hands-on experience withAzure Databricks

ProficiencyinPythonfor data processing

Strong knowledge ofSQL(joins, window functions, performance tuning)

Hands-on experience withApache Spark /PySpark

Experience working withDelta Lake

Knowledge ofAzure Data Lake Storage (ADLS Gen2)

Understanding of distributed computing concepts

Experience with Git version control

Additional Information

Experience withAzure Data Factory

Basic understanding ofdatamodeling

Familiarity withcloud security and RBAC in Azure

Exposure tostreaming data(Spark Structured Streaming, Event Hub, Kafka)