Sr. Databricks Developer

ScatterPie Analytics Private Limited; Pune District; On-site; INR 800,000 - 1,200,000; Full time

Job summary

A data analytics firm in Pune is seeking a Data Engineer to design and optimize data pipelines using Databricks and PySpark. Candidates should have 3-5 years of experience in data engineering with strong skills in SQL and Delta Lake. The role involves collaborative scrum work, troubleshooting data issues, and contributing to architecture discussions. An ideal candidate is proficient in Agile methodologies and has a strong understanding of data governance standards.

Qualifications

• 3-5 years of experience in data engineering, preferably with Azure Databricks.
• Strong technical expertise in PySpark, SQL, and Delta Lake.
• Ability to work independently as well as collaboratively within an agile delivery team.

Responsibilities

• Design, develop, and optimize data pipelines using Databricks and PySpark.
• Collaborate within a scrum team to deliver high-quality, production-ready code.
• Troubleshoot data issues and optimize performance for existing pipelines.

Skills

PySpark

SQL

Delta Lake

Data governance

DevOps practices

Tools

Azure Databricks

ADLS

ADF

Azure Synapse

Job description

Bringing data closer to people. ScatterPie Analytics is a company focused on modernizing data analytics through the implementation of technologies that companies need today to succeed in the market. We bring together the best of open source and blend with the best of enterprise tools to build intelligence that is powerful, cost-effective and flexible.

Job Description

• Design, develop, and optimize data pipelines using Databricks, PySpark, Delta Lake, and related Azure components.
• Work across pipeline development, data modeling, and production code optimization, ensuring scalability and performance.
• Build new pipelines from scratch as well as enhance and maintain existing ones.
• Apply strong understanding of ETL design principles, data modeling concepts, and data governance standards.
• Collaborate within a scrum team, taking ownership of assigned stories while independently delivering high-quality, production-ready code.
• Demonstrate proficiency across PySpark, SQL, Delta Lake, Unity Catalog, and Databricks Workflows, with solid understanding of logic and data flow.
• Work in Azure environments, leveraging tools like ADLS, ADF, and Synapse (as applicable).
• Contribute inputs to architecture and design discussions where appropriate, while primarily focusing on hands-on development.
• Troubleshoot data issues, perform root-cause analysis, and optimize performance for existing pipelines.

Requirements

• 3–5 years of experience in data engineering, preferably with Azure Databricks.
• Strong technical expertise in PySpark, SQL, and Delta Lake.
• Familiarity with Unity Catalog, data governance, and DevOps practices (Git, CI/CD).
• Ability to work independently as well as collaboratively within an agile delivery team.
• Excellent problem-solving, debugging, and communication skills.