DataBricks Technical Lead Exp 8 Yrs To 12 Yrs

Bilvantis Technologies; Hyderabad; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Bilvantis Technologies seeks an experienced DataBricks Technical Lead to drive scalable data pipelines and Lakehouse workflows. You will guide cross-functional teams, review code, and ensure delivery excellence across cloud platforms such as Azure and AWS.

The role requires strong PySpark, Spark SQL, and Databricks expertise, with fluency in data governance, CI/CD, and Agile processes. Leadership, proactive communication, and mentoring are essential.

Qualifications

• 8–12 years IT experience with 3+ years in big data and Databricks.
• Hands-on Databricks, PySpark, Spark SQL expertise.
• Experience with cloud data lakes (Azure AWS) and orchestration tools (ADF/Glu e).
• Strong SQL and relational DB skills; CI/CD and Git experience.
• Knowledge of Delta Lake, Unity Catalog, medallion architecture.
• Streaming data experience with Structured Streaming, Kafka, Event Hubs.
• Terraform for infra as code; Jira/Confluence for agile delivery.
• Excellent communication; leadership and mentoring capabilities.

Responsibilities

• Design scalable, high-performance pipelines and Lakehouse workflows on Databricks.
• Build complex ETL/ELT pipelines on Azure or AWS cloud platforms.
• Guide development teams, perform code reviews, enforce best practices.
• Mentor engineers on Databricks, Spark, and data engineering patterns.
• Collaborate with product owners and architects to translate requirements.
• Monitor production workloads, tune ETL/ELT performance, ensure reliability.

Skills

Databricks

PySpark

Spark SQL

Azure ADLS

AWS S3

Azure Data Factory

AWS Glue

SQL

Git

CI/CD

Terraform

Jira

Confluence

Delta Lake

Unity Catalog

Medallion Architecture

Structured Streaming

Kafka

Event Hubs

Education

Bachelor's or Master's degree in Computer Science or Engineering

Databricks Certified Data Engineer Professional

Microsoft Certified: DP-700 (Fabric Data Engineer) or AWS Data Engineer - Associate

Tools

Azure Data Factory

Azure Databricks

AWS Glue

Azure ADLS

AWS S3

SQL Server / relational databases

Delta Lake

Unity Catalog

Terraform

Kafka

Event Hubs

Job description

Job Description:

Job Description: DataBricks – Technical Lead (Exp: 8 Yrs to 12 Yrs)

We are seeking a talented and motivated DataBricks expert to join our team as a Technical Lead.

Required Skills & Qualifications

• 8 to 12 years of IT experience, with 3+ years focused on big data and Databricks.
• Strong hands-on experience with Databricks, PySpark, and Spark SQL.
• Experience with modern cloud data lakes (Azure ADLS, AWS S3).
• Hands-on experience with Azure Data Factory (ADF) or AWS Glue for orchestration.
• Strong experience with relational databases and SQL.
• Strong background in Agile delivery, CI/CD pipelines, and version control (Git).
• Experience with Delta Lake, Unity Catalog, and the medallion architecture.
• Experience with streaming data (Structured Streaming, Kafka, Event Hubs).
• Experience with Infrastructure-as-Code (Terraform).
• Experience in agile development processes using Jira and Confluence.
• Design scalable data engineering solutions and lead their delivery on Databricks.
• Understanding of the SDLC and Agile methodologies.
• Communication with the customer and producing the daily status report.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Databricks Certified Data Engineer Professional certification.
• Microsoft Certified: Fabric Data Engineer Associate (DP-700) or AWS Certified Data Engineer - Associate.
• Should have good oral and written communication.
• Should be able to lead and mentor a team.
• Should be proactive and adaptive.

Key Responsibilities

• Design scalable, high-performance data pipelines and Lakehouse workflows using Databricks and PySpark / Spark SQL.
• Build complex ETL/ELT pipelines on cloud platforms such as Azure or AWS.
• Guide development teams, perform code reviews, and enforce engineering best practices.
• Mentor engineers on Databricks, Spark, and data engineering design patterns.
• Work with product owners, business analysts, and architects to translate requirements into technical solutions.
• Monitor production workloads, fine-tune ETL/ELT performance, and ensure data reliability.