SENIOR SOFTWARE ENGINEER - Azure Databricks

Happiest Minds Technologies; Bengaluru; Hybrid; INR 2,000,000 - 3,500,000; Full time

Job summary

Happiest Minds Technologies in Bengaluru (Hybrid) is seeking a SaaS Databricks Engineer with 5-8 years of data engineering experience to design, develop, and optimize cloud-native data pipelines on Azure Databricks. You will leverage PySpark, Spark SQL, Delta Lake, and ADF to deliver scalable ETL/ELT workflows for SaaS applications while ensuring data quality, security, and cost efficiency.

The role emphasizes implementing Bronze-Silver-Gold Lakehouse architectures, integrating external data

Qualifications

• 5-8 years of Data Engineering experience.
• 3+ years of hands-on Azure Databricks experience.
• Strong expertise in PySpark and Spark SQL.
• Experience with Azure Data Factory (ADF).
• Knowledge of Delta Lake and Lakehouse Architecture.
• Familiarity with Azure Data Lake Storage Gen2 (ADLS).
• Proficiency in SQL Server / Azure SQL.
• Python programming skills.
• Experience in ETL/ELT development and optimization.
• CI/CD experience using Azure DevOps or GitHub Actions.
• Understanding of Data Modelling and Data Warehousing concepts.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.
• Build and optimize ETL/ELT workflows for large-scale data processing.
• Implement and manage Bronze, Silver, and Gold Lakehouse architectures using Delta Lake.
• Integrate SaaS applications and external data sources into enterprise data platforms.
• Develop data ingestion frameworks using Azure Data Factory (ADF).
• Optimize Databricks clusters, notebooks, and Spark workloads for performance and cost efficiency.
• Implement data quality, validation, audit, and reconciliation controls.
• Work with Azure Data Lake Storage Gen2 (ADLS), SQL databases, and cloud-native storage solutions.
• Automate deployment processes using CI/CD pipelines (Azure DevOps or GitHub Actions).
• Perform troubleshooting, root cause analysis, and production support activities.
• Ensure security, governance, and compliance using RBAC, Key Vault, and managed identities.
• Monitor pipeline health and implement proactive alerting and observability solutions.
• Collaborate with architects, analysts, business stakeholders, and product teams.
• Create technical documentation, data flow diagrams, and operational runbooks.
• Continuously recommend improvements in platform scalability, reliability, and cost optimization.

Skills

PySpark

Spark SQL

Azure Databricks

Azure Data Factory

Delta Lake

ADLS Gen2

SQL Server / Azure SQL

Python

ETL/ELT development

CI/CD (Azure DevOps / GitHub Actions)

Data Modeling

Data Warehousing

Education

Bachelor's degree in Computer Science, Information Technology, or a related field

Tools

Azure DevOps

GitHub Actions

Job description

SaaS Databricks Engineer

Location: Bengaluru (Hybrid)

Experience: 5-8 Years

Employment Type: Full-Time

Job Summary: We are seeking a highly skilled SaaS Databricks Engineer to design, develop, and optimize cloud-native data engineering solutions on Azure Databricks. The ideal candidate will have strong expertise in PySpark, SQL, Delta Lake, Azure Data Factory, and modern data platform architectures. The role involves building scalable ETL/ELT pipelines, supporting SaaS applications, and ensuring reliable data delivery across enterprise environments.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.
• Build and optimize ETL/ELT workflows for large-scale data processing.
• Implement and manage Bronze, Silver, and Gold Lakehouse architectures using Delta Lake.
• Integrate SaaS applications and external data sources into enterprise data platforms.
• Develop data ingestion frameworks using Azure Data Factory (ADF).
• Optimize Databricks clusters, notebooks, and Spark workloads for performance and cost efficiency.
• Implement data quality, validation, audit, and reconciliation controls.
• Work with Azure Data Lake Storage Gen2 (ADLS), SQL databases, and cloud-native storage solutions.
• Automate deployment processes using CI/CD pipelines (Azure DevOps or GitHub Actions).
• Perform troubleshooting, root cause analysis, and production support activities.
• Ensure security, governance, and compliance using RBAC, Key Vault, and managed identities.
• Monitor pipeline health and implement proactive alerting and observability solutions.
• Collaborate with architects, analysts, business stakeholders, and product teams.
• Create technical documentation, data flow diagrams, and operational runbooks.
• Continuously recommend improvements in platform scalability, reliability, and cost optimization.

Mandatory Skills

• 5-8 years of Data Engineering experience.
• 3+ years of hands-on Azure Databricks experience.
• Strong expertise in PySpark and Spark SQL.
• Experience with Azure Data Factory (ADF).
• Knowledge of Delta Lake and Lakehouse Architecture.
• Familiarity with Azure Data Lake Storage Gen2 (ADLS).
• Proficiency in SQL Server / Azure SQL.
• Python programming skills.
• Experience in ETL/ELT development and optimization.
• CI/CD experience using Azure DevOps or GitHub Actions.
• Understanding of Data Modelling and Data Warehousing concepts.

Preferred Skills

• Databricks Certified Data Engineer Associate/Professional.
• Microsoft Azure Data Engineer Associate (DP-203).
• Azure Fundamentals (AZ-900).

Qualifications: Bachelor's degree in Computer Science, Information Technology, or a related field is preferred. Strong analytical and problem-solving skills, along with excellent communication and collaboration abilities, are essential for success in this role.

Azure DevOps