Senior/Lead Data Engineer

ICICI Lombard Group; Mumbai; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

ICICI Lombard Group seeks a senior Data Engineer to design and optimize cloud-based data pipelines using Databricks, PySpark, and Delta Lake. You will architect ingestion for varied data types and collaborate with Data Science to prepare datasets and feature pipelines.

You will implement governance, metadata, and lineage while supporting CI/CD-driven deployments. The role requires hands-on Spark/Databricks expertise, strong SQL skills, and experience with lakehouse architectures; Power BI

Qualifications

• Strong hands-on knowledge of Databricks, Spark, Delta Lake, and PySpark.
• Experience designing scalable cloud pipelines in Azure and SQL expertise.
• Experience with Git, CI/CD, and DevOps workflows; REST APIs experience.
• Knowledge of data warehousing concepts, lakehouse patterns, and metadata management.
• Exposure to ML model deployment workflows.
• Intermediate Power BI experience for datasets, modelling, and dashboards.

Responsibilities

• Develop and optimize data pipelines and ETL/ELT using Databricks.
• Architect data ingestion frameworks for structured, semi-structured, and unstructured data.
• Collaborate with Data Science and Analytics teams; ensure governance and lineage.
• Maintain documentation for pipelines, data flows, and designs.
• Support production deployments and monitoring.

Skills

Databricks

Spark

Delta Lake

PySpark

Azure

SQL

CI/CD

Git

Power BI

ML deployment

Lakehouse patterns

Metadata management

Education

Graduation/Post Graduation

Tools

Databricks

Spark

Delta Lake

PySpark

Unity Catalog

REST APIs

Job description

Responsibilities-

1. Data Engineering & Pipeline Development

Monitor and optimize compute usage across workloads.

Evaluate storage patterns, caching techniques, and data partitioning. Design, build, and maintain scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).

Architect data ingestion frameworks to process structured, semi-structured, and unstructured data.

Develop re-usable frameworks for data quality, data validation, and exception handling.

Data bricks & Cloud Platform Expertise

• Optimize Data bricks clusters, jobs, and queries for performance and cost efficiency.
• Manage cluster configurations, auto scaling, job scheduling, and deployment workflows.
• Implement best practices for notebook modularization, CI/CD, and version control.

Data Architecture & Modelling

• Design end-to-end data models including lakehouse architecture, medallion layers (bronzesilvergold).
• Collaborate with Data Science team to prepare training datasets and feature engineering pipelines.
• Implement governance, cataloguing, and lineage using cloud-native tools.

Performance & Cost Optimization

• Recommend architecture and platform-level improvements to reduce cost.

Collaboration & Documentation

• Work closely with cross-functional teams including Data Science, Analytics, Business, and IT Infra.
• Maintain detailed documentation for pipelines, data flows, and technical designs.
• Support production deployments, monitoring, and troubleshooting.

Educational Qualifications

Graduation/Post Graduation

Competencies Required:

• Strong hands-on knowledge of Databricks, Spark, Delta Lake, and PySpark.
• Experience in architecting and scaling cloud-based pipelines in Azure Strong SQL expertise and understanding of distributed data processing.
• Experience with source versioning (Git), CI/CD, and DevOps workflows. Proficiency with REST APIs, Databricks SQL Warehouses, and Unity Catalog.
• Good understanding of data warehousing concepts, lakehouse patterns, and metadata management.
• Exposure to ML model deployment workflows
• Intermediate proficiency in Power BI (dataset preparation, modeling, dashboards).

Experience -

5+ years of experience as a Data Engineer or in a similar role.