Data Engineering Manager

Crisil; Mumbai; On-site; INR 3,500,000 - 7,500,000; Full time

Job summary

Crisil in Mumbai seeks a Data Engineering Manager to design, build, and optimise a Databricks-based data platform. This hands-on role spans modern data architecture, scalable pipelines, and governance, with migration from legacy systems to Databricks.

The candidate will drive data engineering standards, champion CI/CD, and collaborate with architecture, product and business teams to deliver robust, secure data solutions.

Qualifications

• 7 to 10 years’ data engineering experience with production pipelines.
• Extensive Databricks experience, incl. migration or Lakehouse projects.
• Deep Spark expertise: PySpark, Spark SQL, ideally Scala.
• Strong Delta Lake and Lakehouse design knowledge.
• Proficient in Python and SQL.
• Experience with cloud data platforms, preferably AWS.
• Experience implementing data governance with Unity Catalog.
• IaC experience with Terraform or similar.
• Knowledge of CI/CD, testing, deployment pipelines.
• Understanding data warehousing concepts: Kimball, star/snowflake schemas.
• Experience integrating dbt with modern data platforms.
• Excellent communication and stakeholder skills.
• Experience in scaled Agile or product-led delivery.
• Quality, maintainability, documentation focus.

Responsibilities

• Design, build and optimise scalable data pipelines on Databricks.
• Lead migration from legacy platforms to a Databricks Lakehouse.
• Define data models, quality, performance, and governance standards.
• Collaborate with architecture, product, tech and business teams.
• Institute DevOps/DataOps practices: CI/CD, IaC, versioning, testing.
• Implement Unity Catalog access controls and data lineage.
• Improve cluster performance and cost efficiency.
• Collaborate on data products, analytics, and benchmarking use cases.
• Provide leadership on engineering patterns for ingestion, transformation and release.

Skills

Databricks

Apache Spark

Python

SQL

Delta Lake

Lakehouse design

Data governance

Terraform

CI/CD

dbt

Tools

GitHub Actions

Azure DevOps

AWS

Terraform

Job description

Data Engineering Manager

About the Role

We are seeking a Data Engineering Manager to help design, build, and optimise our next-generation data platform on Databricks. This is a hands‑on engineering role for someone who can work across modern data architecture, scalable data pipelines, platform engineering, and data governance.

You will play a key role in migrating from legacy data platforms to Databricks, working closely with architecture, product, technology, and business teams to turn complex requirements into robust, secure, and scalable data solutions.

The successful candidate will bring deep technical experience in data engineering, strong Databricks capability, and the ability to influence engineering standards and delivery quality across a modern data platform environment.

What the Job Involves

Working as part of the Data Transformation Team, you will be responsible for:

• Designing, developing, and optimising scalable data pipelines using Databricks, Apache Spark, Delta Lake, Python, and SQL.
• Building reliable ETL and ELT processes that support analytics, reporting, benchmarking, and data product use cases.
• Contributing to the design and implementation of the Databricks Lakehouse architecture, including data modelling, data quality, performance, and scalability.
• Supporting the migration of data workloads from legacy platforms to Databricks, working with architects and engineering teams to recommend practical technical solutions.
• Implementing data governance and security controls using Databricks Unity Catalog, including access control, lineage, and environment-level governance.
• Improving the performance and cost efficiency of Databricks clusters, SQL warehouses, workloads, and serverless compute.
• Applying DevOps and DataOps practices, including CI/CD, automated deployment, version control, testing, and infrastructure as code.
• Working with tools such as Terraform, GitHub Actions, Azure DevOps, or equivalent engineering toolchains.
• Collaborating with Data Transformation, Architecture, Product, Technology, and business stakeholders to deliver high-quality data solutions.
• Supporting engineering best practice through code reviews, technical documentation, design input, and knowledge sharing.
• Helping to establish repeatable engineering patterns for data ingestion, transformation, orchestration, testing, monitoring, and release management.

Requirements

We are looking for someone with:

• 7 to 10 years’ experience in data engineering, including strong hands-on experience designing and building production-grade data pipelines.
• Significant experience working with Databricks, ideally in a platform migration, modernisation, or Lakehouse implementation context.
• Deep expertise in Apache Spark, including PySpark, Spark SQL, and ideally Scala.
• Strong knowledge of Delta Lake and Lakehouse design patterns.
• Advanced proficiency in Python and SQL.
• Experience working with cloud data platforms and native cloud data services, ideally AWS.
• Practical experience implementing data governance patterns, ideally using Databricks Unity Catalog.
• Experience with infrastructure as code, particularly Terraform or similar tools.
• Good understanding of CI/CD, automated testing, deployment pipelines, and modern engineering practices.
• Knowledge of data warehousing concepts, including dimensional modelling, Kimball methodology, star schemas, and snowflake schemas.
• Experience integrating dbt or similar transformation frameworks with modern data platforms.
• Strong problem-solving skills and the ability to translate complex technical challenges into clear, practical solutions.
• Excellent communication skills, including the ability to explain technical topics to both engineering and non-technical stakeholders.
• Experience working in scaled Agile or product-led delivery environments.
• A strong focus on quality, maintainability, documentation, and engineering discipline.

Nice to Have

It would be advantageous if you have:

• Databricks Certified Data Engineer Professional certification.
• Databricks Certified Lakehouse Platform Architect certification.
• Familiarity with MLOps or machine learning engineering workflows.
• Experience working within a Product Development Life Cycle.
• Experience supporting distributed teams across time zones.
• Exposure to Jobs-To-Be-Done or Human-Centred Design approaches.
• Industry experience in financial services, benchmarking, market analytics, or data analytics.
• Familiarity with scaled Agile ceremonies such as PI Planning.
• A team-oriented, proactive, analytical working style, with the ability to operate across detailed engineering work and broader platform context.