Lead Databricks Engineer

Blend360 India; Hyderabad; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Blend360 India is seeking an experienced Azure Databricks Engineer to design, build, and optimize scalable data pipelines on the Azure cloud. You will work with Python, PySpark, SQL, and Spark to process large datasets and support analytics and ML use cases.

You will implement ETL/ELT workflows, tune Spark jobs for performance and cost, manage Delta Lake tables, and ensure data quality, security, and governance.

Qualifications

• 6+ years of data engineering experience.
• Hands-on with Azure Databricks and PySpark.
• Strong Python and SQL skills (joins, window functions).
• Experience with Delta Lake and ADLS Gen2.
• Familiarity with Git version control and CI/CD tooling.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Implement ETL/ELT workflows using PySpark, Spark SQL, and Python.
• Optimize Spark jobs for performance, cost, and scalability.
• Work with structured and semi-structured data (Parquet, Delta, JSON, CSV).
• Build and manage Delta Lake tables (ACID, time travel, schema evolution).
• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2).
• Develop complex queries and transformations using SQL.
• Collaborate with data scientists, analysts, and stakeholders to support analytics and ML use cases.
• Ensure data quality, validation, and monitoring.
• Follow best practices for security, access control, and governance in Azure.

Skills

Azure Databricks

Python

SQL

Apache Spark

PySpark

Delta Lake

Azure Data Lake Storage Gen2

Git version control

Azure DevOps

Kafka

Spark Structured Streaming

Tools

Azure Databricks

Git

GitHub Actions

Azure DevOps

Job description

Role & responsibilities

We are looking for an experiencedAzure Databricks Engineerwith strong hands‑on expertise in Python, SQL, and Apache Sparkto design, build, andoptimizescalable data pipelines and analytics solutions on theAzure cloud platform. The ideal candidate should have experience working with large datasets, distributed data processing, and modern data engineering practices.

Responsibilities

• Design, develop, andmaintainscalabledata pipelines using Azure Databricks
• ImplementETL/ELT workflowsusingPySpark, Spark SQL, and Python
• OptimizeSpark jobs forperformance, cost, and scalability
• Work with structured and semi-structured data (Parquet, Delta, JSON, CSV)
• Build and manageDelta Laketables (ACID, time travel, schema evolution)
• Integrate Databricks withAzure Data Lake Storage (ADLS Gen2)
• Develop complex queries and transformations usingSQL
• Collaborate with data scientists, analysts, and stakeholders to support analytics and ML use cases
• Ensure data quality, validation, and monitoring
• Follow best practices for security, access control, and governance in Azure

Preferred candidate profile

• 6+ years of experience inData Engineering
• Strong hands‑on experience withAzure Databricks
• ProficiencyinPythonfor data processing
• Strong knowledge ofSQL(joins, window functions, performance tuning)
• Hands‑on experience withApache Spark /PySpark
• Experience working withDelta Lake
• Knowledge ofAzure Data Lake Storage (ADLS Gen2)
• Understanding of distributed computing concepts
• Experience with Git version control

Additional information

• Experience withAzure Data Factory
• Exposure toCI/CD pipelines(Azure DevOps, GitHub Actions)
• Basic understanding ofdatamodeling
• Familiarity withcloud security and RBAC in Azure
• Exposure to streaming data (Spark Structured Streaming ,Event Hub ,Kafka )