Databricks Engineering Manager

Unison Group; Bengaluru; On-site; INR 3,000,000 - 5,000,000; Full time

Job summary

Unison Group is seeking an experienced Data Engineering Manager to lead a skilled team and drive scalable Lakehouse solutions on Databricks. You will shape engineering standards, build reusable frameworks and ensure secure, governed data products across the organization.

Role requires hands-on proficiency with Spark/PySpark, Delta Lake and cloud-native services, along with strong leadership, collaboration and roadmap delivery capabilities in a fast-paced environment.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, IT, or related field.
• Significant data engineering leadership experience and team management.
• Strong hands-on Databricks and Lakehouse architecture expertise.
• Proficiency in Spark/PySpark, SQL, Python, Delta Lake, ETL/ELT and orchestration.
• Knowledge of Medallion Architecture, distributed processing and cloud data patterns.
• Experience with CI/CD, automated testing, logging, monitoring and controlled deployments.
• Familiarity with data governance, lineage, quality, security and access control.
• Proven ability to optimize large-scale workloads for performance and cost.
• Excellent stakeholder communication and design/code review experience.

Responsibilities

• Lead, mentor and guide a data engineering team; set technical direction.
• Design, build and operate scalable data pipelines and Lakehouse solutions.
• Define standards for ingestion, transformation, testing, deployment and observability.
• Collaborate with business analytics, data science, security and architecture teams.
• Promote reusable, metadata-driven frameworks and modern data engineering practices.
• Embed data quality, governance, privacy and compliance controls.
• Review designs, code and deployment approaches for reliability and maintainability.
• Optimize Databricks workloads for performance, scale and cost efficiency.
• Support Agile delivery, backlog refinement and roadmap development.

Skills

Databricks

Apache Spark / PySpark

SQL

Python

Delta Lake

Medallion Architecture

CI/CD

Data governance

Cloud data-platform patterns

Stakeholder management

Education

Bachelor's or Master's in CS/Engineering/Data Science

Tools

Azure Databricks

Azure DevOps

Git

Terraform / IaC

Unity Catalog

Job description

Role Summary

We are seeking an experienced Data Engineering Manager with strong Databricks and modern cloud data engineering expertise to lead the design, development and operation of scalable enterprise data solutions. The role combines technical leadership, people management, architecture oversight and delivery accountability. The successful candidate will establish engineering standards, reusable frameworks and robust practices for secure, governed, reliable and cost-efficient Lakehouse solutions.

Key Responsibilities

• Lead, mentor and support a team of data engineers, setting technical direction and encouraging engineering best practices.
• Design, build and operate scalable data pipelines and Lakehouse solutions using Databricks, Apache Spark, Delta Lake, SQL, Python and cloud-native services.
• Define standards for ingestion, transformation, orchestration, automated testing, deployment, observability, monitoring and production support.
• Collaborate with business, analytics, data science, architecture, security and platform teams to translate requirements into high-quality data solutions.
• Champion reusable and metadata-driven frameworks, automation, Medallion Architecture patterns and modern data engineering practices.
• Embed data quality, security, governance, privacy and compliance controls across pipelines and data products.
• Review solution designs, code and deployment approaches to maintain consistency, resilience, maintainability and operational reliability.
• Optimize Databricks workloads for performance, scalability, reliability and cost efficiency.
• Support Agile delivery, including backlog refinement, sprint planning, estimation, risk management, continuous improvement and technical roadmap development.

Required Qualifications & Experience

• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, Data Science or a related discipline.
• Significant professional experience in data engineering, including experience leading or managing data engineering teams.
• Strong hands-on experience with Databricks and Lakehouse-based architectures.
• Advanced proficiency in Apache Spark / PySpark, SQL, Python, Delta Lake, ETL/ELT, data pipeline development and orchestration.
• Strong understanding of Medallion Architecture, distributed processing and modern cloud data-platform patterns.
• Experience implementing CI/CD, automated testing, logging, monitoring, alerting and controlled deployment practices.
• Working knowledge of data governance, lineage, data quality, security, privacy and role-based access control.
• Proven experience optimizing large-scale workloads for performance, reliability and cost.
• Experience conducting design and code reviews and establishing engineering standards.
• Strong stakeholder management, problem-solving and written and verbal communication skills.
• Preferred Qualifications
• Experience with Azure Databricks, Unity Catalog and the broader Azure data ecosystem.
• Experience building reusable, configuration-driven or metadata-driven ingestion and transformation frameworks.
• Exposure to Azure DevOps, Git-based CI/CD, infrastructure as code and automated release controls.
• Experience supporting production-grade data platforms and operational support models.
• Understanding of dimensional modelling, enterprise data modelling and semantic or data-serving layers.
• Databricks or relevant cloud certification.
• Experience delivering governed data products for Power BI, analytics, AI/ML or advanced analytics.