Databricks Engineer

Crisil; Maharashtra; On-site; INR 2,500,000 - 4,200,000; Full time

Job summary

CRISIL is seeking an experienced Databricks Engineer to design, develop, and maintain scalable data pipelines on the Databricks Lakehouse platform. You will build ETL/ELT workflows, optimize Spark workloads, and integrate data from multiple enterprise sources.

The role requires strong hands-on experience with Databricks, Apache Spark, PySpark, SQL, Python and Delta Lake, plus cloud platform familiarity. You will collaborate with cross-functional teams in a fast-paced, global environment.

Qualifications

• Bachelor’s degree in Computer Science, Software Engineering or a related field.
• Experience as a Python Developer / SQL with a strong portfolio of projects.
• Strong interpersonal and decision-making skills.
• Ability to work in a global, team-oriented environment with diverse partners.
• Attention to detail and a proactive, execution-focused mindset.
• Multitask effectively in a fast-paced setting with high quality output.
• Willingness to take initiative and drive tasks to completion with strong execution.

Responsibilities

• Design and develop scalable data pipelines using Databricks, Spark and PySpark.
• Build and maintain ETL/ELT pipelines for structured and semi-structured data.
• Develop data transformation workflows using Python, PySpark and SQL.
• Implement and manage Delta Lake tables and related pipelines.
• Use Databricks Workflows/jobs for orchestration and scheduling.
• Integrate data from databases, APIs, files, and cloud storage.
• Implement data quality, validation, reconciliation and monitoring.
• Optimize Spark jobs and queries for performance and cost.
• Apply partitioning, caching and file-management best practices.
• Develop incremental data processing and Change Data Capture (CDC) solutions.
• Collaborate with Data Scientists, Analysts and Tech teams; follow SDLC practices.

Skills

Databricks

PySpark

SQL

Python

Delta Lake

Education

Bachelor’s degree in Computer Science or related field

Tools

Tableau

PowerBI

Azure

AWS

GCP

Job description

CRISIL is a global analytical company providing ratings, research, and risk and policy advisory services. We are India's leading ratings agency. We are also the foremost provider of high-end research to the world's largest banks and leading corporations. CRISIL's majority shareholder is S&P Global.

Division: Global Analytical Centre (GAC)

Role Overview:
We are looking for an experienced Databricks Engineer to design, develop, and maintain scalable data engineering solutions using the Databricks Lakehouse platform. The role will involve building robust ETL/ELT pipelines, developing data transformation frameworks, optimizing Spark workloads, and integrating data from multiple enterprise sources.
The ideal candidate should have strong hands on experience with Databricks, Apache Spark, PySpark, SQL, Python, Delta Lake, and cloud data platforms, along with good understanding of data architecture and data governance.

Key Responsibilities:

• Design and develop scalable data pipelines using Databricks, Apache Spark and PySpark.
• Build and maintain ETL/ELT pipelines for structured and semi-structured data.
• Develop data transformation and processing workflows using Python, PySpark and SQL.
• Implement and manage Delta Lake tables, including Delta Live Tables / Lakeflow pipelines where applicable.
• Work with Databricks Workflows/jobs for pipeline orchestration and scheduling.
• Integrate data from databases, APIs, files, cloud storage and other enterprise systems.
• Implement data quality, validation, reconciliation and monitoring frameworks.
• Optimize Spark jobs, SQL queries and Databricks workloads for performance and cost.
• Implement appropriate partitioning, caching, clustering and file-management strategies.
• Develop incremental data processing and Change Data Capture (CDC) solutions.
• Work with cloud platforms such as Azure, AWS or GCP.
• Implement security, access controls and data governance in collaboration with platform teams.
• Troubleshoot production data pipelines and resolve performance and data-quality issues.
• Collaborate with Data Scientists, Business Analysts, Application teams and Technology teams.
• Participate in code reviews, technical design discussions and deployment activities.
• Follow Agile development practices and enterprise SDLC standards.
• Maintain technical documentation, data lineage and operational runbooks.
  

Required Technical Skills:

• Strong hands-on experience with Databricks
• PySpark

Qualification and Essential Skills:

• Bachelor’s degree in Computer Science, Software Engineering or a related field.
• Experience as a Python Developer / SQL with a strong portfolio of projects.
• Good interpersonal and decision-making skills.
• Ability to work effectively in a team-oriented, global environment with various business partners.
• Well-organized with great attention to detail.
• Ability to multitask and stay poised in a fast-paced, high-pressure environment while also ensuring the highest quality output.
• Flexible, self-starter who is willing to take the initiative and drive tasks to completion with strong execution.
• Must possess strong prioritization and influencing skills, experience in leading highly valued projects.

Eligibility Criteria:

• Not appeared for CRISIL test / interviews in the last six months
• Familiar with Tableau, and PowerBI is required.
• Certification in Databricks, Cloud technologies, AWS, Agile Methodology and Project Management will be added advantage.