Technical Lead - Data Engineer ( Databricks Azure)

Srijan: Now Material; Gurugram District; On-site; INR 2,400,000 - 4,800,000; Full time

Job summary

Srijan: Now Material is seeking a skilled Data Engineer to lead the design, development, and optimization of scalable data pipelines on Azure. The role focuses on ingesting data from Microsoft platforms, SAP, CRM, and retail sources into a centralized data platform to support analytics and AI initiatives.

The ideal candidate will implement Medallion Architecture data lakes and leverage Azure-native services to ensure near real-time data availability, quality, and governance for downstream

Qualifications

• 5+ years of data engineering experience with at least 2+ years in a lead role.
• Hands-on Azure data services expertise and modern data lake architectures.
• Experience building Medallion Architecture data lakes (Bronze/Silver/Gold).
• Proficiency in PySpark and large-scale data processing.
• Familiarity with CI/CD pipelines and data governance practices.

Responsibilities

• Design and implement scalable ETL/ELT pipelines ingesting data from enterprise systems into Azure-based data platforms.
• Design, build, and maintain Medallion Architecture Data Lakes (Bronze, Silver, Gold) using Azure technologies.
• Develop and optimize data processing workflows with Azure Databricks, Spark, PySpark, Data Factory, and related services.
• Create analytics-ready data models aligned with downstream apps, microservices, and AI usage.
• Implement idempotent processing, CDC upsert, and incremental processing for reliable pipelines.
• Leverage Azure Data Lake Storage Gen2, Azure SQL, Synapse, Cosmos DB, and related services for storage and querying.
• Implement data ingestion patterns using Event Hubs, Functions, Logic Apps, Service Bus, and Data Factory.
• Lead enterprise data integration across Microsoft platforms, SAP, SaaS apps, APIs, and external sources.
• Apply performance tuning to improve pipeline efficiency, reliability, and cost-effectiveness.
• Define data governance, quality, metadata management, and data lineage standards.
• Collaborate with DevOps to maintain CI/CD pipelines using Azure DevOps or GitHub Actions.
• Support AI and agent-enabled solutions leveraging Azure AI Foundry, OpenAI, and related technologies.
• Provide technical leadership and mentorship to the data engineering team.
• Coordinate with application, analytics, and AI teams to ensure seamless data platform integration via APIs and data products.

Skills

Azure data services

Medallion Architecture

PySpark

Data lake architecture

ETL/ELT design

CI/CD pipelines

Data governance

Tools

Azure Databricks

Azure Data Factory

Azure Synapse Analytics

Cosmos DB

Job description

Data Engineering

We are seeking a highly skilled and experienced

Data Engineering

We are seeking a highly skilled and experienced Data Engineering with strong expertise in Microsoft Azure data services, enterprise data integration, and modern data lake architectures. In this role, you will lead the design, development, and optimization of scalable data pipelines responsible for ingesting and transforming data from enterprise systems including Microsoft platforms, SAP, CRM systems, retail systems, and external third-party data sources into a centralized data platform to support downstream applications, analytics, AI, and agentic use cases.

You will play a critical role in building a robust, high-performance data platform using Azure-native services, ensuring data quality, reliability, and near real-time availability for business operations. The ideal candidate will have hands-on experience building Medallion Architecture (Bronze, Silver, Gold) data lakes, integrating diverse enterprise data sources, and enabling AI-powered solutions using the Microsoft ecosystem.

Responsibilities

• Design and implement scalable ETL/ELT pipelines to ingest data from enterprise systems including Microsoft applications, SAP, CRM platforms, retail systems, and external third-party sources into Azure-based data platforms.


• Design, build, and maintain enterprise-scale Medallion Architecture Data Lakes (Bronze, Silver, Gold) using Azure Data Lake Storage, Microsoft Fabric, Databricks, or similar technologies.


• Develop and optimize data processing workflows using Azure Databricks, Apache Spark / PySpark, Azure Data Factory, Microsoft Fabric, and other Azure-native services.


• Create denormalized, API-ready, analytics-ready, and AI-ready data models aligned with downstream applications, microservices, reporting, and agentic AI consumption patterns.


• Implement idempotent processing, CDC merge (upsert) strategies, incremental processing, and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.


• Leverage Azure Data Lake Storage Gen2, Azure SQL, Azure Synapse Analytics, Microsoft Fabric, Cosmos DB, and related services for efficient storage and querying of structured and semi-structured data.


• Implement data ingestion patterns (batch, streaming, and event-driven) using tools such as Azure Event Hubs, Azure Functions, Logic Apps, Service Bus, and Azure Data Factory.


• Lead enterprise data integration initiatives across Microsoft platforms, SAP, SaaS applications, APIs, partner systems, and external data providers.


• Apply performance tuning and optimization techniques to improve pipeline efficiency, scalability, reliability, and cost-effectiveness.


• Define and enforce data governance, data quality, metadata management, and data lineage standards across the data platform.


• Collaborate with DevOps teams to design and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools.


• Support AI and agent-enabled solutions leveraging Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, and related Microsoft AI technologies.


• Conduct peer reviews and provide technical leadership and mentorship to the data engineering team.


• Collaborate with application, microservices, analytics, and AI teams to ensure seamless integration with enterprise data platforms via APIs, event streams, and data products.

Requirements

• 5+ years of experience in data engineering, with at least 2+ years in a lead role.


• Strong hands-on experience with Azure data services such as Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse Analytics, Microsoft Fabric, and Azure SQL.


• Proven experience designing and implementing Medallion Architecture Data Lakes (Bronze, Silver, Gold) and modern lakehouse solutions.


• Proficiency in PySpark / Spark for large-scale data processing and optimization.


• Experience designing and implementing ODS, Lakehouse, or operational data layers using technologies such as Cosmos DB, Azure SQL, Microsoft Fabric, Databricks, or similar platforms.


• Strong expertise in ETL/ELT design patterns, enterprise data integration, data ingestion, and transformation pipelines.


• Experience integrating data from Microsoft platforms, SAP systems, CRM applications, REST APIs, SaaS applications, and external third-party data sources.


• Hands-on experience with CI/CD pipelines (Azure DevOps, GitHub Actions, or similar).


• Good understanding of data governance, data quality, metadata management, and data lineage frameworks.


• Experience supporting microservices, analytics platforms, AI solutions, and agentic architectures with enterprise data platforms.


• Exposure to Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, or related Microsoft AI technologies is highly desirable.


• Strong problem-solving skills and ability to optimize complex data workflows.


• Excellent communication and stakeholder management skills.


• Ability to work in a fast-paced, agile environment and manage multiple priorities.