Senior Databricks Engineer

Judge India Solutions; Pune District; On-site; INR 3,500,000 - 5,500,000; Full time

Benefits offered by this job

null

Job summary

Judge India Solutions looks for a Senior Databricks Data Engineer to provide technical leadership in a large-scale BI modernization initiative. The role drives the transformation from a Microsoft BI ecosystem to a scalable Databricks Lakehouse, guiding architecture, standards, and governance across teams.

You will lead migration of a ~20TB data warehouse, design Spark-based frameworks, and collaborate with BI, analytics, and AI/ML teams to enable advanced analytics.

Qualifications

• Bachelor’s or master’s degree in computer science, engineering, or related field.
• 8–12 years of data engineering experience.
• 5+ years of hands-on experience building and operating production workloads on Databricks.
• Deep expertise in SQL, PySpark, and distributed data processing.
• Demonstrated leadership in structured enterprise BI modernization programs.
• Proven experience replacing SSIS-based ETL with scalable Spark-based frameworks.
• Strong understanding of SSAS cube architecture and semantic layer modernization.
• Experience designing and operating high-volume, enterprise-grade analytical data platforms.
• Strong knowledge of Delta Lake, Lakehouse and Medallion architectures, data versioning, and Workflows/Jobs orchestration.
• Experience implementing CI/CD, testing frameworks, and operational reliability for data platforms.
• AWS-based Databricks experience strongly preferred.

Responsibilities

• Lead the architecture and implementation of enterprise-scale data pipelines on Databricks.
• Serve as technical lead for migration of ~20TB SQL Server data warehouse to Databricks Lakehouse.
• Design and implement reusable Spark-based pipeline frameworks to replace SSIS workflows.
• Deconstruct SSAS cube logic and redesign into Lakehouse-native semantic models.
• Replace legacy SSRS reporting ecosystems with modern, governed analytics layers.
• Rebuild dimensional models within Delta Lake using scalable Lakehouse design patterns.
• Define data modeling standards, ingestion frameworks, CI/CD processes, and operational best practices.
• Implement and enforce Medallion architecture standards with performance optimization and governance.
• Optimize Spark workloads for scalability, reliability, and cost efficiency.
• Implement governance, lineage, security, and monitoring using Unity Catalog and tooling.
• Mentor engineers, conduct architectural reviews, establish and maintain engineering standards.
• Partner with BI, analytics, and AI/ML teams to enable advanced analytics and downstream use cases.
• Influence platform roadmap and architectural decisions.

Skills

Leadership

BI modernization

SQL

PySpark

Data pipelines

CI/CD

Spark tuning

Databricks

Delta Lake

Education

Bachelorf's degree in CS/Engineering

Master's degree in CS/Engineering

Tools

Databricks

Unity Catalog

Delta Lake

Job description

Rel. Experience: 10+ Years

We are seeking a Senior Databricks Data Engineer to provide technical leadership for a large-scale enterprise BI modernization initiative. This is not a lift-and-shift migration effort. The role will lead the systematic transformation of a complex Microsoft BI ecosystem (SSIS, SSAS, SSRS) into scalable, governed Databricks Lakehouse architecture.

Key Responsibilities

• Lead the architecture and implementation of enterprise-scale data pipelines on Databricks.
• Serve as technical lead for the migration of a ~20TB Microsoft SQL Server data warehouse to the Databricks Lakehouse Platform.
• Design and implement reusable Spark-based pipeline frameworks to replace thousands of SSIS workflows.
• Deconstruct SSAS cube logic and redesign it into Lakehouse-native semantic models.
• Replace legacy SSRS reporting ecosystems with modern, governed analytics layers.
• Rebuild dimensional models within Delta Lake using scalable Lakehouse design patterns.
• Define data modeling standards, ingestion frameworks, CI/CD processes, and operational best practices.
• Implement and enforce Medallion architecture standards (Bronze/Silver/Gold) with performance optimization and governance controls.
• Optimize Spark workloads for scalability, reliability, and cost efficiency.
• Implement governance, lineage, security, and monitoring using Unity Catalog and related platform tooling.
• Mentor engineers, conduct architectural reviews, establish and maintain engineering standards.
• Partner with BI, analytics, and AI/ML teams to enable advanced analytics and downstream use cases.
• Influence platform roadmap and architectural decisions.

Required Qualifications

• Bachelor’s or master’s degree in computer science, engineering, or related field.
• 8 – 12 years of data engineering experience.
• 5+ years of hands-on experience building and operating production workloads on Databricks.
• Deep expertise in SQL, PySpark, and distributed data processing.
• Demonstrated leadership in structured enterprise BI modernization programs.
• Proven experience replacing SSIS-based ETL with scalable Spark-based frameworks.
• Strong understanding of SSAS cube architecture and semantic layer modernization.
• Experience designing and operating high-volume, enterprise-grade analytical data platforms.
• Strong knowledge of Delta Lake, Lakehouse and Medallion architectures, data versioning, and Workflows/Jobs orchestration.
• Experience implementing CI/CD, testing frameworks, and operational reliability for data platforms.
• AWS-based Databricks experience strongly preferred.

Preferred Skills

• Experience designing semantic layers and BI integrations (Power BI preferred).
• Exposure to Databricks AI/ML capabilities (MLflow, feature engineering, feature stores).
• Experience establishing enterprise migration patterns across distributed teams.
• Prior technical leadership experience in global or multi-team environments.
• Strong stakeholder engagement and communication skills.