Technical Lead - Data Engineer ( Databricks Azure)

Srijan Technologies PVT LTD; Gurugram District; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Srijan Technologies PVT LTD is hiring a Data Engineer to design, build and optimize scalable data pipelines on the Azure platform. The role requires leading data lake structures (Bronze, Silver, Gold) and integrating diverse enterprise sources including SAP, CRM, and external data feeds.

You will implement robust data models, facilitate AI-ready data products, and ensure near real-time data availability for analytics, reporting, and AI initiatives in a fast-paced environment.

Qualifications

• 5+ years of experience in data engineering, with at least 2+ years in a lead role.
• Strong hands-on experience with Azure data services such as Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse Analytics, Microsoft Fabric, and Azure SQL.
• Proven experience designing and implementing Medallion Architecture Data Lakes (Bronze, Silver, Gold) and modern lakehouse solutions.
• Proficiency in PySpark / Spark for large-scale data processing and optimization.
• Experience designing and implementing ODS, Lakehouse, or operational data layers using Cosmos DB, Azure SQL, Microsoft Fabric, Databricks, or similar platforms.
• Strong expertise in ETL/ELT design patterns, enterprise data integration, data ingestion, and transformation pipelines.
• Experience integrating data from Microsoft platforms, SAP systems, CRM applications, REST APIs, SaaS applications, and external third-party data sources.
• Hands-on experience with CI/CD pipelines (Azure DevOps, GitHub Actions, or similar).
• Good understanding of data governance, data quality, metadata management, and data lineage frameworks.
• Experience supporting microservices, analytics platforms, AI solutions, and agentic architectures with enterprise data platforms.
• Exposure to Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, or related Microsoft AI technologies is highly desirable.
• Strong problem-solving skills and ability to optimize complex data workflows.
• Excellent communication and stakeholder management skills.
• Ability to work in a fast-paced, agile environment and manage multiple priorities.

Responsibilities

• Design and implement scalable ETL/ELT pipelines to ingest data from enterprise systems into Azure-based data platforms.
• Design, build, and maintain Medallion Architecture Data Lakes (Bronze, Silver, Gold) using Azure Data Lake Storage Gen2, Microsoft Fabric, Databricks, or similar technologies.
• Develop and optimize data processing workflows using Azure Databricks, Spark / PySpark, Azure Data Factory, Microsoft Fabric, and other Azure-native services.
• Create denormalized, API-ready, analytics-ready, and AI-ready data models aligned with downstream applications and AI consumption patterns.
• Implement idempotent processing, CDC merge (upsert) strategies, incremental processing, and data reconciliation mechanisms.
• Leverage Azure data services for efficient storage and querying of structured and semi-structured data.
• Implement data ingestion patterns using Azure Event Hubs, Azure Functions, Logic Apps, Service Bus, and Azure Data Factory.
• Lead enterprise data integration initiatives across Microsoft platforms, SAP, SaaS apps, APIs, and external data providers.
• Apply performance tuning to improve pipeline efficiency, scalability, reliability, and cost-effectiveness.
• Define and enforce data governance, data quality, metadata management, and data lineage standards.
• Collaborate with DevOps to design and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or similar.
• Support AI and agent-enabled solutions leveraging Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, and related technologies.
• Conduct peer reviews and provide technical leadership to the data engineering team.
• Collaborate with application, microservices, analytics, and AI teams to ensure seamless integration via APIs, event streams, and data products.

Skills

Azure data services

ETL/ELT design patterns

PySpark / Spark

CI/CD

Data governance

stakeholder management

problem solving

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Azure Databricks

Azure Synapse Analytics

Microsoft Fabric

Cosmos DB

Azure SQL

Databricks

GitHub Actions

Azure DevOps

Job description

Data Engineering

We are seeking a highly skilled and experienced Data Engineeringwith strong expertise in Microsoft Azure data services, enterprise data integration, and modern data lake architectures. In this role, you will lead the design, development, and optimization of scalable data pipelines responsible for ingesting and transforming data from enterprise systems including Microsoft platforms, SAP, CRM systems, retail systems, and external third-party data sources into a centralized data platform to support downstream applications, analytics, AI, and agentic use cases.

You will play a critical role in building a robust, high-performance data platform using Azure-native services, ensuring data quality, reliability, and near real-time availability for business operations. The ideal candidate will have hands‑on experience building Medallion Architecture (Bronze, Silver, Gold) data lakes, integrating diverse enterprise data sources, and enabling AI‑powered solutions using the Microsoft ecosystem.

Responsibilities

• Design and implement scalable ETL/ELT pipelines to ingest data from enterprise systems including Microsoft applications, SAP, CRM platforms, retail systems, and external third‑party sources into Azure‑based data platforms.
• Design, build, and maintain enterprise‑scale Medallion Architecture Data Lakes (Bronze, Silver, Gold) using Azure Data Lake Storage, Microsoft Fabric, Databricks, or similar technologies.
• Develop and optimize data processing workflows using Azure Databricks, Apache Spark / PySpark, Azure Data Factory, Microsoft Fabric, and other Azure‑native services.
• Create denormalized, API‑ready, analytics‑ready, and AI‑ready data models aligned with downstream applications, microservices, reporting, and agentic AI consumption patterns.
• Implement idempotent processing, CDC merge (upsert) strategies, incremental processing, and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.
• Leverage Azure Data Lake Storage Gen2, Azure SQL, Azure Synapse Analytics, Microsoft Fabric, Cosmos DB, and related services for efficient storage and querying of structured and semi‑structured data.
• Implement data ingestion patterns (batch, streaming, and event‑driven) using tools such as Azure Event Hubs, Azure Functions, Logic Apps, Service Bus, and Azure Data Factory.
• Lead enterprise data integration initiatives across Microsoft platforms, SAP, SaaS applications, APIs, partner systems, and external data providers.
• Apply performance tuning and optimization techniques to improve pipeline efficiency, scalability, reliability, and cost‑effectiveness.
• Define and enforce data governance, data quality, metadata management, and data lineage standards across the data platform.
• Collaborate with DevOps teams to design and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools.
• Support AI and agent‑enabled solutions leveraging Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, and related Microsoft AI technologies.
• Conduct peer reviews and provide technical leadership and mentorship to the data engineering team.
• Collaborate with application, microservices, analytics, and AI teams to ensure seamless integration with enterprise data platforms via APIs, event streams, and data products.

Requirements

• 5+ years of experience in data engineering, with at least 2+ years in a lead role.
• Strong hands‑on experience with Azure data services such as Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse Analytics, Microsoft Fabric, and Azure SQL.
• Proven experience designing and implementing Medallion Architecture Data Lakes (Bronze, Silver, Gold) and modern lakehouse solutions.
• Proficiency in PySpark / Spark for large‑scale data processing and optimization.
• Experience designing and implementing ODS, Lakehouse, or operational data layers using technologies such as Cosmos DB, Azure SQL, Microsoft Fabric, Databricks, or similar platforms.
• Strong expertise in ETL/ELT design patterns, enterprise data integration, data ingestion, and transformation pipelines.
• Experience integrating data from Microsoft platforms, SAP systems, CRM applications, REST APIs, SaaS applications, and external third‑party data sources.
• Hands‑on experience with CI/CD pipelines (Azure DevOps, GitHub Actions, or similar).
• Good understanding of data governance, data quality, metadata management, and data lineage frameworks.
• Experience supporting microservices, analytics platforms, AI solutions, and agentic architectures with enterprise data platforms.
• Exposure to Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, or related Microsoft AI technologies is highly desirable.
• Strong problem‑solving skills and ability to optimize complex data workflows.
• Excellent communication and stakeholder management skills.
• Ability to work in a fast‑paced, agile environment and manage multiple priorities.