Databricks Lead – Delivery & Engineering

Jade Global Software Pvt LTD; Pune District; On-site; INR 223,200 - 334,800; Full time

Job summary

Jade Global Software Pvt LTD is looking for a Databricks Lead to manage hands-on delivery and optimize data engineering practices. You will lead engineering teams while ensuring adherence to best practice standards.

The ideal candidate has over 10 years of experience in data engineering and 3–5 years specifically with Databricks, along with strong skills in Apache Spark and Delta Lake.

This position emphasizes leadership in building robust data platforms and optimizing performance, crucial for the success of data-driven initiatives.

Qualifications

• Lead hands-on Databricks delivery, ensuring adherence to CoE standards and best practices.
• Deep expertise in Databricks, Apache Spark, and Delta Lake.
• Strong hands-on experience with PySpark and Spark SQL.

Responsibilities

• Lead Databricks engineering teams.
• Build and optimize Spark & Delta Lake pipelines.
• Implement Databricks Jobs, Workflows, and CI/CD.
• Ensure reliability, scalability, and observability of data pipelines.

Skills

Databricks

Apache Spark

Delta Lake

PySpark

Spark SQL

Performance tuning

Data modeling

Education

10–12 years in data engineering

3–5 years hands-on Databricks experience

Tools

Databricks Data Engineer Associate

Databricks Data Engineer Professional

Job description

Databricks Lead – Delivery & Engineering

Lead hands‑on Databricks delivery, ensuring adherence to CoE standards and best practices.

Key Responsibilities

• Lead Databricks engineering teams
• Build and optimize Spark & Delta Lake pipelines
• Implement Databricks Jobs, Workflows, CI/CD
• Optimize performance and DBU usage
• Contribute to Databricks accelerators and standards
• Design and implement end‑to‑end data platforms using Databricks Lakehouse
• Lead development of:
  • Data ingestion (batch & streaming)
  • Delta Lake data modeling (Bronze/Silver/Gold)
  • Scalable ETL/ELT pipelines
• Optimize Spark workloads for performance and cost
• Ensure reliability, scalability, and observability of data pipelines

Experience

• 10–12 years in data engineering
• 3–5 years hands‑on Databricks experience

Skills and Knowledge

• Deep expertise in Databricks, Apache Spark, Delta Lake
• Strong hands‑on experience with PySpark / Spark SQL (core, SQL, streaming)
• Experience designing Lakehouse architectures
• Strong understanding of data modeling, data quality, and lineage
• Databricks platform features
• Cloud‑native data engineering
• Performance tuning & troubleshooting

Preferred Certifications

• Databricks Data Engineer Associate
• Databricks Data Engineer Professional