Azure Data Engineer

Orcapod Consulting Services; Mumbai; Hybrid; INR 3,000,000 - 5,400,000; Full time

Job summary

Orcapod Consulting Services seeks an experienced Azure Data Engineer to design, develop, and optimize scalable data platforms on Azure, with emphasis on Databricks, PySpark, Delta Lake, and Lakehouse architectures.

You will build and maintain ETL/ELT pipelines using Azure Data Factory and Databricks, design scalable ingestion and data quality frameworks, and implement governance using Unity Catalog or Purview. Collaboration with data scientists and business stakeholders is essential.

Qualifications

• 5+ years of Data Engineering experience.
• Strong hands-on experience with Azure Databricks, PySpark, Python, and SQL.
• Experience with Azure Data Factory (ADF), ADLS Gen2, and Azure Synapse.
• Expertise in Delta Lake, Lakehouse, and Medallion Architecture.
• Good understanding of ETL/ELT, Data Warehousing, and Data Modeling.
• Experience with data governance tools like Unity Catalog or Microsoft Purview.

Responsibilities

• Develop and maintain ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Build and optimize Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Design scalable data ingestion, transformation, and data quality frameworks.
• Work with structured and semi-structured data formats such as Parquet, Avro, JSON, and ORC.
• Implement data governance, lineage, and access controls using Unity Catalog and/or Microsoft Purview.
• Support AI-ready data products, RAG pipelines, and vector search use cases.
• Monitor, troubleshoot, and optimize pipeline performance and cloud costs.
• Collaborate with Data Scientists, AI Engineers, Architects, and business stakeholders.

Skills

Azure Databricks

PySpark

Python

SQL

Azure Data Factory

ADLS Gen2

Azure Synapse

Delta Lake

Lakehouse Architecture

Medallion Architecture

Data Warehousing

Data Modeling

Unity Catalog / Microsoft Purview

Tools

dbt

Great Expectations

Delta Live Tables

Terraform

Bicep

Azure DevOps

GitHub Actions

Job description

We are looking for an experienced Azure Data Engineer to design, develop, and optimize scalable data platforms and pipelines on Azure. The ideal candidate should have strong expertise in Azure Databricks, PySpark, Azure Data Factory, ADLS Gen2, and Delta Lake, with experience building modern Lakehouse architectures.

Key Responsibilities

• Develop and maintain ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Build and optimize Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Design scalable data ingestion, transformation, and data quality frameworks.
• Work with structured and semi-structured data formats such as Parquet, Avro, JSON, and ORC.
• Implement data governance, lineage, and access controls using Unity Catalog and/or Microsoft Purview.
• Support AI-ready data products, RAG pipelines, and vector search use cases.
• Monitor, troubleshoot, and optimize pipeline performance and cloud costs.
• Collaborate with Data Scientists, AI Engineers, Architects, and business stakeholders.

Required Skills

• 5+ years of Data Engineering experience.
• Strong hands-on experience with Azure Databricks, PySpark, Python, and SQL.
• Experience with Azure Data Factory (ADF), ADLS Gen2, and Azure Synapse.
• Expertise in Delta Lake, Lakehouse, and Medallion Architecture.
• Good understanding of ETL/ELT, Data Warehousing, and Data Modeling.
• Experience with data governance tools like Unity Catalog or Microsoft Purview.

Preferred Skills

• Azure AI Search, RAG, Vector Search, Embeddings.
• Streaming technologies such as Kafka, Event Hubs, or Structured Streaming.
• dbt, Great Expectations, Delta Live Tables.
• Terraform, Bicep, Azure DevOps, GitHub Actions.