Assistant Manager - Databricks

Adani Group; Ahmedabad District; On-site; INR 1,500,000 - 2,100,000; Full time

Job summary

Adani Group is seeking a Databricks Engineer to lead the design, development and operations of scalable data pipelines and Lakehouse solutions on Databricks. You will translate business requirements into reliable, secure, governed data solutions.

You will work across teams to optimize performance, cost, and security while ensuring production readiness. 3–5 years of hands-on Databricks experience is expected, with strong Python/PySpark and SQL skills.

Qualifications

• Strong hands-on Databricks development experience.
• Good Python and PySpark coding skills.
• Good SQL and data modelling knowledge.
• Experience with Delta Lake, Databricks Jobs/Workloads and performance optimisation.
• Knowledge of data ingestion, transformation, orchestration and production support practices.

Responsibilities

• Lead Databricks-based data engineering delivery for batch, streaming and analytical use cases.
• Design and implement data pipelines using Python, PySpark, SQL, Delta Lake and Databricks workflows.
• Implement data quality, error handling, logging, lineage, metadata and operational monitoring controls.
• Work with architects to define Lakehouse patterns, medallion architecture, reusable frameworks and data product standards.
• Support Unity Catalog, access control, data governance, security and compliance implementation.
• Optimise Databricks jobs, clusters, SQL queries and storage patterns for performance and cost.
• Coordinate with business, analytics, AI, cloud, cybersecurity and operations teams to deliver production-grade solutions.
• Review code, guide engineers, resolve technical blockers and ensure delivery quality.
• Maintain documentation, design artefacts, runbooks and operational handover material.

Skills

Databricks development

Python

PySpark

SQL

Data modelling

Delta Lake

Databricks Jobs/Workflows

CI/CD

Security

Tools

GitHub

Azure DevOps

Databricks Asset Bundles

Job description

Purpose/Objective

Databricks Engineer will lead the design, development and operations of scalable data pipelines and Lakehouse solutions on Databricks. The role will be responsible for translating business and platform requirements into reliable, secure, governed and cost-efficient data solutions.

Key Responsibilities of Role

• Key Responsibilities - Lead Databricks-based data engineering delivery for batch, streaming and analytical use cases.
• - Design and implement data pipelines using Python, PySpark, SQL, Delta Lake and Databricks workflows.
• - Implement data quality, error handling, logging, lineage, metadata and operational monitoring controls.
• - Work with architects to define Lakehouse patterns, medallion architecture, reusable frameworks and data product standards.
• - Support Unity Catalog, access control, data governance, security and compliance implementation.
• - Optimise Databricks jobs, clusters, SQL queries and storage patterns for performance and cost.
• - Coordinate with business, analytics, AI, cloud, cybersecurity and operations teams to deliver production-grade solutions.
• - Review code, guide engineers, resolve technical blockers and ensure delivery quality.
• - Maintain documentation, design artefacts, runbooks and operational handover material.
• Required Skills - Strong hands-on Databricks development experience.
• - Strong Python and PySpark coding skills.
• - Good SQL and data modelling knowledge.
• - Experience with Delta Lake, Databricks Jobs/Workflows and performance optimisation.
• - Knowledge of data ingestion, transformation, orchestration and production support practices.
• - Understanding of cloud storage, IAM, networking and security basics for data platforms.
• - Ability to lead small teams and coordinate delivery across multiple stakeholders.
• Preferred Skills - Unity Catalog and Databricks governance experience.
• - Delta Live Tables, structured streaming and real-time data processing exposure.
• - CI/CD for data engineering using GitHub, Azure DevOps or Databricks Asset Bundles.
• - Knowledge of data mesh, data products, data quality frameworks and MLOps/DataOps.
• - Databricks certification will be an added advantage.
• Success Measures - Stable, reusable and well-governed Databricks pipelines in production.
• - Improved data quality, lineage, auditability and platform reliability.
• - Reduction in pipeline failures, rework and manual operational intervention.
• - Improved performance and cost efficiency of Databricks workloads.
• - Better adoption of standardised Lakehouse and data engineering patterns.

Technical Competencies

Databricks & Lakehouse Architecture,Data Engineering & Pipeline Development,Data Governance, Security & Operational Excellence,Performance Optimisation & Platform Engineering

Qualifications and Experience

3-5 years overall experience, with strong hands-on Databricks and data engineering background