Technical Lead - Data Engineer

Srijan: Now Material; Gurugram District; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Srijan: Now Material is seeking a Data Engineer to design, develop and optimize scalable data pipelines on Azure, enabling real-time access to analytics and AI-ready data. You will lead data integration across Microsoft platforms, SAP and external sources, implementing Medallion Architecture and modern lakehouse patterns.

The role requires strong PySpark, Azure Databricks, Data Factory and data governance experience, with leadership responsibilities in a fast-paced environment.

Qualifications

• 5+ years of data engineering experience with at least 2+ years in a lead role.
• Hands-on experience with Azure data services and modern lakehouse solutions.
• Proficiency in PySpark / Spark for large-scale data processing.
• Experience designing Medallion Architecture Data Lakes (Bronze, Silver, Gold).
• Experience with CI/CD pipelines and data governance practices.

Responsibilities

• Design and implement scalable ETL/ELT pipelines into Azure-based data platforms.
• Build and maintain Medallion Architecture Data Lakes (Bronze, Silver, Gold).
• Develop data processing workflows using Azure Databricks, Spark, PySpark, and Azure Data Factory.
• Create analytics-ready data models for downstream applications and AI consumption.
• Lead data integration initiatives across Microsoft platforms, SAP, and external sources.
• Ensure data quality, governance, and lineage across pipelines.

Skills

Azure data services

PySpark / Spark

ETL/ELT design

Data ingestion

Data governance

Stakeholder management

Agile / fast-paced environments

Azure Databricks

Azure Synapse Analytics

Education

Bachelor's degree in Computer Science or related field

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Azure Cosmos DB

Microsoft Fabric

Databricks

Azure SQL

Azure Event Hubs

Azure DevOps

GitHub Actions

Job description

Job Description:

Data Engineering

We are seeking a highly skilled and experienced Data Engineeringwith strong expertise in Microsoft Azure data services, enterprise data integration, and modern data lake architectures. In this role, you will lead the design, development, and optimization of scalable data pipelines responsible for ingesting and transforming data from enterprise systems including Microsoft platforms, SAP, CRM systems, retail systems, and external third-party data sources into a centralized data platform to support downstream applications, analytics, AI, and agentic use cases.

You will play a critical role in building a robust, high-performance data platform using Azure-native services, ensuring data quality, reliability, and near real-time availability for business operations. The ideal candidate will have hands‑on experience building Medallion Architecture (Bronze, Silver, Gold) data lakes, integrating diverse enterprise data sources, and enabling AI-powered solutions using the Microsoft ecosystem.

Responsibilities

• Design and implement scalable ETL/ELT pipelines to ingest data from enterprise systems including Microsoft applications, SAP, CRM platforms, retail systems, and external third-party sources into Azure-based data platforms.
• Design, build, and maintain enterprise-scale Medallion Architecture Data Lakes (Bronze, Silver, Gold) using Azure Data Lake Storage, Microsoft Fabric, Databricks, or similar technologies.
• Develop and optimize data processing workflows using Azure Databricks, Apache Spark / PySpark, Azure Data Factory, Microsoft Fabric, and other Azure-native services.
• Create denormalized, API-ready, analytics-ready, and AI-ready data models aligned with downstream applications, microservices, reporting, and agentic AI consumption patterns.
• Implement idempotent processing, CDC merge (upsert) strategies, incremental processing, and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.
• Leverage Azure Data Lake Storage Gen2, Azure SQL, Azure Synapse Analytics, Microsoft Fabric, Cosmos DB, and related services for efficient storage and querying of structured and semi-structured data.
• Implement data ingestion patterns (batch, streaming, and event-driven) using tools such as Azure Event Hubs, Azure Functions, Logic Apps, Service Bus, and Azure Data Factory.
• Lead enterprise data integration initiatives across Microsoft platforms, SAP, SaaS applications, APIs, partner systems, and external data providers.
• Apply performance tuning and optimization techniques to improve pipeline efficiency, scalability, reliability, and cost-effectiveness.
• Define and enforce data governance, data quality, metadata management, and data lineage standards across the data platform.
• Collaborate with DevOps teams to design and maintain CI/CD pipelines using Azure DevOps, GitHub Actions, or similar tools.
• Support AI and agent-enabled solutions leveraging Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, and related Microsoft AI technologies.
• Conduct peer reviews and provide technical leadership and mentorship to the data engineering team.
• Collaborate with application, microservices, analytics, and AI teams to ensure seamless integration with enterprise data platforms via APIs, event streams, and data products.

Requirements

• 5+ years of experience in data engineering, with at least 2+ years in a lead role.
• Strong hands‑on experience with Azure data services such as Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, Azure Synapse Analytics, Microsoft Fabric, and Azure SQL.
• Proven experience designing and implementing Medallion Architecture Data Lakes (Bronze, Silver, Gold) and modern lakehouse solutions.
• Proficiency in PySpark / Spark for large-scale data processing and optimization.
• Experience designing and implementing ODS, Lakehouse, or operational data layers using technologies such as Cosmos DB, Azure SQL, Microsoft Fabric, Databricks, or similar platforms.
• Strong expertise in ETL/ELT design patterns, enterprise data integration, data ingestion, and transformation pipelines.
• Experience integrating data from Microsoft platforms, SAP systems, CRM applications, REST APIs, SaaS applications, and external third-party data sources.
• Hands‑on experience with CI/CD pipelines (Azure DevOps, GitHub Actions, or similar).
• Good understanding of data governance, data quality, metadata management, and data lineage frameworks.
• Experience supporting microservices, analytics platforms, AI solutions, and agentic architectures with enterprise data platforms.
• Exposure to Azure AI Foundry, Azure OpenAI Service, Microsoft Copilot Studio, or related Microsoft AI technologies is highly desirable.
• Strong problem‑solving skills and ability to optimize complex data workflows.
• Excellent communication and stakeholder management skills.
• Ability to work in a fast‑paced, agile environment and manage multiple priorities.