DataBricks – Technical Lead (Exp: 8 Yrs to 12 Yrs)

Keka Inc.; Hyderabad; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

DataBricks is seeking a Technical Lead with 8–12 years of experience to design and deliver scalable data pipelines and Lakehouse workflows on cloud platforms. Lead a delivery team, review code, and mentor engineers while coordinating with product owners and architects.

You will drive ETL/ELT solutions on Databricks with PySpark, Spark SQL, Delta Lake, and streaming data, ensuring reliability, performance, and adherence to SDLC and Agile practices.

Qualifications

• 8 to 12 years IT experience, with 3+ years focused on big data and Databricks.
• Strong hands-on experience with Databricks, PySpark, and Spark SQL.
• Experience with modern cloud data lakes (Azure ADLS, AWS S3).

Responsibilities

• Design scalable, high-performance data pipelines and Lakehouse workflows using Databricks and PySpark / Spark SQL.
• Build complex ETL/ELT pipelines on cloud platforms such as Azure or AWS.
• Guide development teams, perform code reviews, and enforce engineering best practices.
• Mentor engineers on Databricks, Spark, and data engineering design patterns.
• Work with product owners, business analysts, and architects to translate requirements into technical solutions.
• Monitor production workloads, fine-tune ETL/ELT performance, and ensure data reliability.

Skills

Databricks

PySpark

Spark SQL

Azure

AWS

ETL/ELT

CI/CD

Git

Terraform

Delta Lake

Structured Streaming

Kafka

Java/Scala

Jira

Confluence

Mentoring

Education

Bachelor's degree in Computer Science/Engineering

Master's degree (preferred)

Tools

Azure Data Factory

AWS Glue

Databricks

Jira

Confluence

Terraform

Job description

DataBricks – Technical Lead (Exp: 8 Yrs to 12 Yrs)

8 Yrs to 12 Yrs

Full-Time

Job Description: DataBricks – Technical Lead (Exp: 8 Yrs to 12 Yrs)

Required Skills & Qualifications:

• 8 to 12 years of IT experience, with 3+ years focused on big data and Databricks.
• Strong hands-on experience with Databricks, PySpark, and Spark SQL.
• Experience with modern cloud data lakes (Azure ADLS, AWS S3).
• Hands-on experience with Azure Data Factory (ADF) or AWS Glue for orchestration.
• Strong experience with relational databases and SQL.
• Strong background in Agile delivery, CI/CD pipelines, and version control (Git).
• Experience with Delta Lake, Unity Catalog, and the medallion architecture.
• Experience with streaming data (Structured Streaming, Kafka, Event Hubs).
• Experience with Infrastructure-as-Code (Terraform).
• Experience in agile development processes using Jira and Confluence.
• Design scalable data engineering solutions and lead their delivery on Databricks.
• Understanding of the SDLC and Agile methodologies.
• Communication with the customer and producing the daily status report.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
• Databricks Certified Data Engineer Professional certification.
• Should have good oral and written communication.
• Should be able to lead and mentor a team.
• Should be proactive and adaptive.

Key Responsibilities:

• Design scalable, high-performance data pipelines and Lakehouse workflows using Databricks and PySpark / Spark SQL.
• Build complex ETL/ELT pipelines on cloud platforms such as Azure or AWS.
• Guide development teams, perform code reviews, and enforce engineering best practices.
• Mentor engineers on Databricks, Spark, and data engineering design patterns.
• Work with product owners, business analysts, and architects to translate requirements into technical solutions.
• Monitor production workloads, fine-tune ETL/ELT performance, and ensure data reliability.