Azure Databricks Engineer

Sonyo Management Consultants; Karnataka; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Sonyo Management Consultants in Bangalore seeks an Azure Databricks Engineer with 4-5 years of data engineering experience to build scalable data pipelines and analytics solutions.

The role requires hands-on skills with Apache Spark, PySpark, Delta Lake, and Azure data services, plus collaboration with data scientists and business stakeholders.

Mentor junior engineers and contribute to best practices while ensuring data security and governance across platforms.

Qualifications

• 4-5 years of experience in data engineering.
• Strong proficiency in Spark, PySpark, and SQL.
• Hands-on experience with Azure Databricks, Azure Data Lake, Data Factory, and Synapse.
• Solid understanding of Delta Lake, Parquet, and other big data formats.
• Familiarity with CI/CD pipelines and DevOps practices in data engineering.
• Excellent problem‑solving and communication skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, strong in Python, PySpark and Apache Spark.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
• Optimize Spark jobs for performance and cost-efficiency in cloud environments.
• Integrate data from various sources including Azure Data Lake, Azure SQL, Blob Storage, and external APIs.
• Implement data quality checks, monitoring, and alerting mechanisms.
• Work with Delta Lake for ACID‑compliant data storage and efficient data versioning.
• Automate workflows using Azure Data Factory, Airflow, or similar orchestration tools.
• Ensure security, governance, and compliance across data platforms.
• Mentor junior engineers and contribute to best practices and standards.

Skills

Python

PySpark

Apache Spark

SQL

Azure Databricks

Azure Data Lake

Azure Data Factory

Azure Synapse Analytics

Delta Lake

Parquet

Airflow

CI/CD pipelines

DevOps practices

Data quality

Tools

Azure Databricks

Azure Data Lake

Azure Data Factory

Azure Synapse Analytics

Job description

Azure Databricks Engineer

Location: Bangalore

Experience: 6-9 Y

Job Summary

We are seeking a highly skilled and experienced Azure Databricks Engineer to join our data engineering team. The ideal candidate will have deep expertise in building scalable data pipelines, optimizing big data workloads, and leveraging Azure cloud services to deliver high-performance analytics solutions. This role requires strong hands‑on experience with Apache Spark, Databricks, and Azure Data Services, along with a solid understanding of data architecture and engineering best practices.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, strong in Python, PySpark and Apache Spark.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver robust solutions.
• Optimize Spark jobs for performance and cost-efficiency in cloud environments.
• Integrate data from various sources including Azure Data Lake, Azure SQL, Blob Storage, and external APIs.
• Implement data quality checks, monitoring, and alerting mechanisms.
• Work with Delta Lake for ACID‑compliant data storage and efficient data versioning.
• Automate workflows using Azure Data Factory, Airflow, or similar orchestration tools.
• Ensure security, governance, and compliance across data platforms.
• Mentor junior engineers and contribute to best practices and standards.

Required Skills & Qualifications

• 4-5 years of experience in data engineering.
• Strong proficiency in Apache Spark, PySpark, and SQL.
• Hands-on experience with Azure Databricks, Azure Data Lake, Azure Data Factory, and Azure Synapse Analytics.
• Solid understanding of Delta Lake, Parquet, and other big data formats.
• Familiarity with CI/CD pipelines and DevOps practices in data engineering.
• Excellent problem‑solving and communication skills.