Azure Databricks Engineer

Orcapod Consulting Services; Hyderabad, Chennai, Bengaluru; Hybrid; INR 2,500,000 - 4,500,000; Full time

Job summary

Orcapod Consulting Services in Hyderabad is seeking an experienced Azure Databricks Engineer to design, develop, and optimize scalable data pipelines and lake solutions using Azure Databricks, PySpark, ADF, ADLS Gen2, and Synapse.

The role emphasizes performance tuning, data quality, governance, and collaboration with data architects and stakeholders to meet business needs.

Qualifications

• 5+ years of experience with Apache Spark / PySpark.
• 5+ years with Azure Databricks and Azure Data Services (ADLS Gen2, ADF, Synapse).
• Strong understanding of data warehousing, ETL, and data lake architectures.

Responsibilities

• Design and implement ETL/ELT pipelines using Azure Databricks and PySpark.
• Develop scalable data ingestion, transformation, and processing solutions.
• Work with structured and unstructured data from ADLS Gen2, SQL databases, APIs, and cloud platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Build and automate workflows using Azure Data Factory (ADF).
• Implement data quality checks, monitoring, logging, and alerting frameworks.
• Collaborate with data architects, analysts, and business stakeholders to gather requirements.
• Participate in code reviews and follow software engineering best practices.
• Ensure data governance, security, and compliance standards are met.

Skills

Azure Databricks

PySpark

Apache Spark

Azure Data Factory

ADLS Gen2

Azure Synapse Analytics

Python

SQL

ETL/ELT Development

Data Warehousing

Data Lake Architecture

Spark Performance Tuning

Tools

Delta Lake

Medallion Architecture

Azure DevOps / CI-CD

Unity Catalog

Job description

Job Summary

We are seeking an experienced Azure Databricks Engineer with strong expertise in Azure Data Engineering technologies. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data lake solutions using Azure Databricks, PySpark, Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.

Key Responsibilities

• Design and implement ETL/ELT pipelines using Azure Databricks and PySpark.
• Develop scalable data ingestion, transformation, and processing solutions.
• Work with structured and unstructured data from various sources such as ADLS Gen2, SQL Databases, APIs, and cloud platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Build and automate workflows using Azure Data Factory (ADF).
• Implement data quality checks, monitoring, logging, and alerting frameworks.
• Collaborate with data architects, analysts, and business stakeholders to gather requirements.
• Participate in code reviews and follow software engineering best practices.
• Ensure data governance, security, and compliance standards are met.

Mandatory Skills

• Azure Databricks
• PySpark
• Apache Spark
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure Synapse Analytics
• Python
• SQL
• ETL/ELT Development
• Data Warehousing
• Data Lake Architecture
• Spark Performance Tuning

Preferred Skills

• Delta Lake
• Medallion Architecture
• Azure DevOps / CI-CD
• Unity Catalog
• Data Quality & Monitoring Frameworks

Required Skills

• 5+ years of experience in Apache Spark / PySpark.
• 5+ years working with Azure Databricks and Azure Data Services (ADLS Gen2, ADF, Synapse).
• Strong understanding of data warehousing, ETL, and data lake architectures