SENIOR SOFTWARE ENGINEER - Azure Databricks

Happiest Minds Technologies; Bengaluru; Hybrid; INR 2,500,000 - 4,000,000; Full time

Job summary

Happiest Minds Technologies in Bengaluru is seeking a SaaS Databricks Engineer to design, develop, and optimize cloud-native data engineering solutions on Azure Databricks. You will work with PySpark, SQL, Delta Lake, and Azure Data Factory to build scalable ETL/ELT pipelines and ensure reliable data delivery across enterprise environments.

The role involves optimizing Databricks clusters, implementing data quality controls, supporting SaaS applications, and collaborating with architects,

Qualifications

• Bachelor's degree preferred in CS/IT or related field.
• 5+ years of data engineering experience in cloud environments.
• Strong proficiency in PySpark and Spark SQL.
• Experience with ETL/ELT pipelines and data lake architectures.
• Hands-on with Azure Databricks, ADF, Delta Lake, and ADLS Gen2.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and optimize ETL/ELT workflows for large-scale data processing.
• Implement and manage Bronze, Silver, and Gold Lakehouse architectures with Delta Lake.
• Integrate SaaS applications and external data sources into enterprise data platforms.
• Develop data ingestion using Azure Data Factory and CI/CD pipelines.
• Optimize Databricks clusters, notebooks, and Spark workloads for performance and cost.
• Ensure data quality, validation, and governance controls.

Skills

Data engineering

Analytical skills

Communication

Education

Bachelor's degree in CS/IT

Tools

Azure Databricks

PySpark

Spark SQL

Azure Data Factory

Delta Lake

ADLS Gen2

SQL Server / Azure SQL

Python

CI/CD (Azure DevOps / GitHub Actions)

Job description

SaaS Databricks Engineer

Location: Bengaluru (Hybrid)

Experience: 5-8 Years

Employment Type: Full-Time

Job Summary: We are seeking a highly skilled SaaS Databricks Engineer to design, develop, and optimize cloud-native data engineering solutions on Azure Databricks. The ideal candidate will have strong expertise in PySpark, SQL, Delta Lake, Azure Data Factory, and modern data platform architectures. The role involves building scalable ETL/ELT pipelines, supporting SaaS applications, and ensuring reliable data delivery across enterprise environments.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.
• Build and optimize ETL/ELT workflows for large-scale data processing.
• Implement and manage Bronze, Silver, and Gold Lakehouse architectures using Delta Lake.
• Integrate SaaS applications and external data sources into enterprise data platforms.
• Develop data ingestion frameworks using Azure Data Factory (ADF).
• Optimize Databricks clusters, notebooks, and Spark workloads for performance and cost efficiency.
• Implement data quality, validation, audit, and reconciliation controls.
• Work with Azure Data Lake Storage Gen2 (ADLS), SQL databases, and cloud-native storage solutions.
• Automate deployment processes using CI/CD pipelines (Azure DevOps or GitHub Actions).
• Perform troubleshooting, root cause analysis, and production support activities.
• Ensure security, governance, and compliance using RBAC, Key Vault, and managed identities.
• Monitor pipeline health and implement proactive alerting and observability solutions.
• Collaborate with architects, analysts, business stakeholders, and product teams.
• Create technical documentation, data flow diagrams, and operational runbooks.
• Continuously recommend improvements in platform scalability, reliability, and cost optimization.

Mandatory Skills

• 5-8 years of Data Engineering experience.
• 3+ years of hands-on Azure Databricks experience.
• Strong expertise in PySpark and Spark SQL.
• Experience with Azure Data Factory (ADF).
• Knowledge of Delta Lake and Lakehouse Architecture.
• Familiarity with Azure Data Lake Storage Gen2 (ADLS).
• Proficiency in SQL Server / Azure SQL.
• Python programming skills.
• Experience in ETL/ELT development and optimization.
• CI/CD experience using Azure DevOps or GitHub Actions.
• Understanding of Data Modelling and Data Warehousing concepts.

Preferred Skills

• Databricks Certified Data Engineer Associate/Professional.
• Microsoft Azure Data Engineer Associate (DP-203).
• Azure Fundamentals (AZ-900).

Qualifications:

Bachelor's degree in Computer Science, Information Technology, or a related field is preferred. Strong analytical and problem-solving skills, along with excellent communication and collaboration abilities, are essential for success in this role.

Azure DevOps