Lead data Engineer

Infojini Inc; Bengaluru; On-site; INR 4,000,000 - 8,000,000; Full time

Job summary

Infojini Inc. seeks a Senior Data Engineering Lead in Bengaluru to drive enterprise-scale data initiatives with hands-on coding in Azure Databricks and PySpark.

You will mentor engineers, lead architecture discussions, and ensure scalable ETL/ELT pipelines and lakehouse architectures with strong governance. You will collaborate on design reviews, optimize Spark workloads, and implement CI/CD, security, and cost controls while delivering high-quality data solutions for complex business

Qualifications

• 5+ years of hands-on experience with Azure Databricks and PySpark.
• Experience with Microsoft Fabric.
• Experience leading teams, architecture discussions, and enterprise data transformation programs.
• Hands-on coding is mandatory; contribute to development, write and optimize production-grade code.

Responsibilities

• Data Engineering Leadership: Lead design, development, and delivery of enterprise-scale data solutions, mentoring data engineers and driving technical excellence.
• Advanced Databricks Development: Strong hands-on expertise in Azure Databricks, PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
• ETL/ELT Design and Implementation: Design, develop, and optimize scalable ETL/ELT pipelines, incremental data ingestion, CDC, and batch/streaming data processing.
• SQL Development and Optimization: Expert-level SQL skills, including complex queries, stored procedures, execution plan analysis, indexing, partitioning, and query performance tuning.
• Azure Data Platform Expertise: Strong experience with Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, and related Azure data services.
• Microsoft Fabric: Good hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Factory, and Spark-based data engineering capabilities.
• Data Modeling and Architecture: Expertise in data warehousing, dimensional modeling, star/snowflake schemas, lakehouse architecture, and data quality management.
• Performance and Cost Optimization: Identify and resolve data processing bottlenecks, optimize Spark workloads, and improve pipeline performance, reliability, and cloud cost efficiency.
• Technical Design and Team Mentoring: Lead technical design discussions, architecture reviews, code reviews, engineering best practices, and knowledge-sharing initiatives.
• Enterprise Integration and DevOps: Experience delivering enterprise-scale data integration and analytics solutions, implementing CI/CD, automated testing, monitoring, governance, security, and production support processes.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Medallion Architecture

Azure Data Factory

Azure Data Lake Storage Gen2

Microsoft Fabric

Data modeling

Lakehouse architecture

Tools

Azure Databricks

Azure Data Factory

Microsoft Fabric

OneLake

Job description

5+ years of hands-on experience in Azure Databricks/PySpark.

Experience with Microsoft Fabric.

Experience leading teams, architecture discussions, and enterprise data transformation programs.

Exposure to DevOps, governance, security, and cloud cost optimization.

Hands-on coding is mandatory for this role. The Senior Data Engineering Lead must actively contribute to development, write and optimize production-grade code, troubleshoot complex technical issues, and lead by example rather than focusing solely on team management or architecture.

• Data Engineering Leadership: Lead the design, development, and delivery of enterprise-scale data engineering solutions, mentoring data engineers and driving technical excellence.
• Advanced Databricks Development: Strong hands-on expertise in Azure Databricks, PySpark, Spark SQL, Delta Lake, and Medallion Architecture.
• ETL/ELT Design and Implementation: Design, develop, and optimize scalable ETL/ELT pipelines, incremental data ingestion, CDC, and batch/streaming data processing.
• SQL Development and Optimization: Expert-level SQL skills, including complex queries, stored procedures, execution plan analysis, indexing, partitioning, and query performance tuning.
• Azure Data Platform Expertise: Strong experience with Azure Data Factory, Azure Data Lake Storage Gen2, Azure Databricks, and related Azure data services.
• Microsoft Fabric: Good hands-on experience with Microsoft Fabric, including OneLake, Lakehouse, Warehouse, Data Factory, and Spark-based data engineering capabilities.
• Data Modeling and Architecture: Expertise in data warehousing, dimensional modeling, star/snowflake schemas, lakehouse architecture, and data quality management.
• Performance and Cost Optimization: Identify and resolve data processing bottlenecks, optimize Spark workloads, and improve pipeline performance, reliability, and cloud cost efficiency.
• Technical Design and Team Mentoring: Lead technical design discussions, architecture reviews, code reviews, engineering best practices, and knowledge-sharing initiatives.
• Enterprise Integration and DevOps: Experience delivering enterprise-scale data integration and analytics solutions, implementing CI/CD, automated testing, monitoring, governance, security, and production support processes.