Senior Data Engineer

Kanini Software Solutions; Pune District, Chennai, Bengaluru; Hybrid; INR 3,500,000 - 7,000,000; Full time

Job summary

Kanini Software Solutions is seeking an experienced Senior Data Engineer in Pune, India. You will design, build, and optimize scalable data pipelines using Azure Databricks, PySpark, SQL, and Azure Data Factory to support business analytics.

The role requires 8–10 years in data engineering, with hands-on Azure cloud skills, data integration expertise, and a focus on performance and reliability. Microsoft Fabric experience is a plus.

Qualifications

• 8–10 years of data engineering experience.
• Hands-on with Azure Databricks for data processing and transformation.
• Proficient in PySpark and SQL with performance tuning.
• Experience with Azure Data Factory (ADF) and data integration.
• Knowledge of Microsoft Fabric is a plus.
• Experience working with data lakes and data warehouses.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, SQL, and ADF.
• Build and optimize ETL/ELT workflows for large data volumes.
• Develop data transformation and integration solutions from multiple sources.
• Write complex SQL queries and optimize for performance.
• Develop PySpark applications for distributed data processing.

Skills

Azure Databricks

PySpark

SQL

Azure Data Factory (ADF)

Microsoft Fabric

Data pipelines design

Tools

Delta Lake

Job description

Job Summary

We are looking for an experienced Senior Data Engineer with strong expertise in Azure, Databricks, PySpark, SQL, and Azure Data Factory (ADF). The ideal candidate should have hands‑on experience designing, developing, and optimizing scalable data pipelines and ETL/ELT workflows, along with integrating data from multiple sources.

Experience with Microsoft Fabric is an added advantage. The candidate should be proficient in data engineering, best practices, performance optimization, and building reliable data solutions to support business and analytical requirements.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, SQL, and Azure Data Factory (ADF).
• Build and optimize ETL/ELT workflows to process large volumes of structured and semi-structured data.
• Develop data transformation and integration solutions to consolidate data from multiple sources.
• Write complex SQL queries and optimize them for performance and efficiency.
• Develop PySpark applications for distributed data processing and large‑scale data transformations.
• Work with Azure cloud services to build secure, scalable, and reliable data engineering solutions.
• Implement data validation, quality checks, error handling, and monitoring mechanisms across data pipelines.
• Troubleshoot pipeline failures and resolve data processing and integration issues.
• Optimize Databricks workloads, Spark jobs, and SQL queries to improve performance and manage costs.
• Collaborate with data architects, data analysts, data scientists, and other stakeholders to understand business requirements and deliver effective data solutions.
• Follow best practices for data security, governance, version control, and deployment.
• Explore and implement Microsoft Fabric capabilities for data integration, engineering, and analytics wherever applicable.

Required Skills and Qualifications

• 8-10 years of overall experience in data engineering.
• Strong hands‑on experience with Azure Databricks for data processing and transformation.
• Proficiency in PySpark for distributed data processing and large‑scale data transformations.
• Strong knowledge of SQL, including complex queries, joins, aggregations, and performance tuning.
• Hands‑on experience with Microsoft Azure data services.
• Experience developing and maintaining ETL/ELT pipelines using Azure Data Factory (ADF).
• Strong understanding of data integration concepts and experience integrating data from multiple sources.
• Experience working with data lakes, data warehouses, and scalable data processing architectures.
• Knowledge of data quality, pipeline monitoring, troubleshooting, and performance optimization.
• Strong analytical, problem‑solving, and communication skills.

Good to Have

• Hands‑on experience with Microsoft Fabric, including Fabric Data Factory, Lakehouse, or other relevant Fabric components.
• Knowledge of Microsoft Fabric integration with Azure Databricks and other Azure data services.
• Experience with Delta Lake and lakehouse architecture.
• Familiarity with CI/CD practices and version control tools such as Git and Azure DevOps.
• Understanding of data governance, access control, and security best practices.