Data Engineering Lead

Kadel Labs; Bengaluru; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

Kadel Labs – Startup Studio seeks a Lead Data Engineer to design and implement scalable data pipelines and analytics platforms in Bengaluru. You will own data modeling, ETL/ELT workflows, and AI-enabled development strategies to accelerate delivery.

You’ll mentor a small team of engineers, collaborate with product owners, data scientists, and analysts, and build Power BI datasets for self-service reporting.

Qualifications

• 8–10 years of data engineering experience.
• Strong Python (PySpark) and SQL skills.
• Experience with Microsoft Fabric and SQL Server.
• Experience leading a small team of engineers.
• Ability to communicate complex ideas to diverse audiences.

Responsibilities

• Design and build scalable data pipelines and ETL/ELT workflows.
• Develop data models using SQL Server and MongoDB.
• Build data solutions on Microsoft Fabric (Lakehouses, Warehouses, Dataflows, Pipelines).
• Lead AI-enabled development to accelerate pipeline work.
• Mentor and grow a team of data engineers.
• Collaborate with stakeholders to translate data needs into solutions.
• Create Power BI data models and datasets for self-service analytics.
• Optimize pipelines for performance, cost, reliability.
• Enforce data engineering standards, security, and governance.

Skills

Python (PySpark)

SQL

Microsoft Fabric

SQL Server

MongoDB Atlas

ETL/ELT pipelines

Power BI

AI-enabled development

Team leadership

Data modeling concepts

Communication

Tools

Databricks

Azure

GCP

AWS

Apache Kafka

Azure Event Hubs

Azure Service Bus

.NET

MLOps tooling

Job description

At Kadel Labs - Startup Studio we built an ecosystem for early-stage founders to help them take their ideas to the next level. The ecosystem consists of in-house product teams, engineers, designers, data scientists, sales & marketing teams, legal experts, taxation, finance and operation heads. We aim to build a cohesive ecosystem, so that your startup runs as a full-fledged business from day 1!

Job Description

Role: Lead Data Engineer

Responsibilities

• Design and build scalable, reliable data pipelines and ETL/ELT workflows across batch and near-real-time processing scenarios
• Develop and maintain data models across relational (SQL Server) and NoSQL (MongoDB, Atlas) systems to support analytical and operational use cases
• Build and manage data solutions on Microsoft Fabric - including Lakehouses, Warehouses, Dataflows, and Pipelines - to deliver a unified analytics platform
• Drive AI-enabled development practices within the team - leveraging AI coding assistants, LLM-based tools, and intelligent automation to accelerate pipeline development, improve code quality, and reduce manual effort
• Lead, mentor, and grow a team of data engineers; conduct code reviews, provide technical guidance, and support career development
• Collaborate with stakeholders, product owners, data scientists, and analysts to understand data needs and translate them into engineering solutions
• Build and maintain Power BI data models, semantic layers, and datasets to support self-service analytics and business reporting
• Continuously identify opportunities to optimize pipeline performance, reduce costs, and improve data reliability and quality
• Adhere to and enforce data engineering standards, data security, and governance practices across the platform

Required Skills and Experience

• 8 to 10 years of experience in data engineering with a strong track record of delivering production-grade data solutions
• Strong proficiency in Python (PySpark, Pandas) and SQL for data transformation, pipeline development, and performance tuning
• Proficient with Microsoft Fabric including Lakehouses, Warehouses, Dataflows Gen2, and Data Pipelines
• Strong experience with SQL Server including schema design, stored procedures, indexing, and query optimization
• Experience with MongoDB and MongoDB Atlas for NoSQL data modeling, indexing, aggregation pipelines, and Atlas Search
• Solid experience designing and operating ETL/ELT pipelines in production, including error handling, monitoring, and SLA management
• Experience with Power BI including dataset design, DAX, semantic modeling, and enabling self-service reporting
• Strong exposure to AI-enabled development - using AI coding assistants, prompt-driven development, or LLM-integrated tooling to build and accelerate data engineering workflows
• Experience leading or managing a small team of engineers - task allocation, mentoring, and performance support
• Good understanding of data modeling concepts - dimensional modeling, star/snowflake schemas, data vault
• Ability to communicate technical ideas clearly to both technical and non-technical audiences

Nice to Have Qualities & Skills

• Hands-on experience with Databricks including Delta Lake, notebooks, jobs, clusters, and Unity Catalog
• Exposure to cloud data services on Azure (preferred), GCP, or AWS
• Experience with streaming and event-driven architectures using Apache Kafka, Azure Event Hubs, Azure Service Bus, or similar queue/messaging technologies
• Exposure to .NET for building data-adjacent services or APIs
• Familiarity with data governance, data cataloging, and data lineage tooling
• Exposure to MLOps or supporting ML pipeline infrastructure