Senior/Lead Data Engineer

ICICI Lombard; Mumbai; On-site; INR 2,800,000 - 4,000,000; Full time

Job summary

ICICI Lombard in Mumbai seeks a seasoned Data Engineer to design, build, and maintain scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake). You will shape ingestion, governance, and analytics-ready data across lakehouse architectures while collaborating with Data Science and Analytics teams.

Ideal candidates have 5+ years in data engineering, strong SQL, Azure experience, and proficiency with CI/CD, version control, and REST APIs.

Qualifications

• Graduate or postgraduate with data engineering focus.
• Strong hands-on knowledge of Databricks, Spark, Delta Lake, and PySpark.
• Experience with architecting cloud-based pipelines in Azure and data warehousing concepts.
• Experience with source control (Git), CI/CD, and DevOps workflows.
• Familiarity with REST APIs, Unity Catalog, and Power BI.

Responsibilities

• Data engineering and pipeline development for scalable ETL/ELT.
• Monitor compute usage and optimize storage, partitioning, and caching.
• Architect ingestion frameworks for structured, semi-structured, and unstructured data.
• Develop data quality, validation, and exception-handling frameworks.
• Optimize Databricks clusters, jobs, and queries for performance and cost.
• Collaborate with Data Science, Analytics, Business, and IT Infra; document pipelines and designs.
• Support deployments, monitoring, and troubleshooting.

Skills

Databricks

Spark

Delta Lake

PySpark

SQL

Git

CI/CD

DevOps

REST APIs

Unity Catalog

Power BI

ML deployment workflows

Azure

Education

Graduation/Post Graduation

Tools

Databricks

Azure

Unity Catalog

Power BI

Job description

Responsibilities-

1. Data Engineering & Pipeline Development

   Monitor and optimize compute usage across workloads.

   • Evaluate storage patterns, caching techniques, and data partitioning.
   • Design, build, and maintain scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).
   • Architect data ingestion frameworks to process structured, semi-structured, and unstructured data.
   • Develop re-usable frameworks for data quality, data validation, and exception handling.
2. Databricks & Cloud Platform Expertise
   • Optimize Databricks clusters, jobs, and queries for performance and cost efficiency.
   • Manage cluster configurations, auto scaling, job scheduling, and deployment workflows.
   • Implement best practices for notebook modularization, CI/CD, and version control.
3. Data Architecture & Modelling
   • Design end-to-end data models including lakehouse architecture, medallion layers (bronzesilvergold).
   • Collaborate with Data Science team to prepare training datasets and feature engineering pipelines.
   • Implement governance, cataloguing, and lineage using cloud-native tools.
4. Performance & Cost Optimization
   • Recommend architecture and platform-level improvements to reduce cost.
5. Collaboration & Documentation
   • Work closely with cross-functional teams including Data Science, Analytics, Business, and IT Infra.
   • Maintain detailed documentation for pipelines, data flows, and technical designs.
   • Support production deployments, monitoring, and troubleshooting.

Educational Qualifications

Graduation/Post Graduation

Competencies Required: –

• Strong hands-on knowledge of Databricks, Spark, Delta Lake, and PySpark.
• Experience in architecting and scaling cloud-based pipelines in Azure
• Strong SQL expertise and understanding of distributed data processing.
• Experience with source versioning (Git), CI/CD, and DevOps workflows.
• Proficiency with REST APIs, Databricks SQL Warehouses, and Unity Catalog.
• Good understanding of data warehousing concepts, lakehouse patterns, and metadata management.
• Exposure to ML model deployment workflows
• Intermediate proficiency in Power BI (dataset preparation, modeling, dashboards).

Experience -

5+ years of experience as a Data Engineer or in a similar role.