Lead Data Engineer, Databricks

Jobtailor; Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Jobtailor in Bengaluru is seeking an experienced Data Engineer to design, build, and maintain scalable data pipelines using Databricks and PySpark. You will shape data lakehouse architectures and optimize ETL/ELT processes across Bronze–Silver–Gold layers.

The role requires 5–10 years in data engineering, with 2–4 years hands-on Databricks experience, and strong SQL/Data warehousing knowledge. You will ensure data quality, governance, and production readiness in a cloud-based environment.

Qualifications

• 5–10 years of experience in Data Engineering.
• Minimum 2–4 years of hands-on Databricks experience.
• Proven experience building enterprise-scale data platforms and pipelines.
• Strong experience in Databricks (Workflows, Notebooks, Jobs).
• PySpark / Spark SQL
• SQL & Data Warehousing concepts
• Hands-on experience with Delta Lake / Lakehouse architecture
• Data pipeline development & orchestration

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
• Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers).
• Develop data ingestion frameworks from multiple sources (batch & real-time).
• Implement Delta Lake features including schema evolution, ACID transactions, and optimization.
• Create and manage Databricks notebooks, workflows, and jobs for production pipelines.
• Ensure data quality, governance, and lineage across pipelines.
• Optimize Spark jobs for performance (partitioning, caching, tuning).
• Collaborate with cross-functional teams (Data Scientists, Analysts, Architects).
• Support production deployments, monitoring, and troubleshooting of data pipelines.
• Contribute to migration of legacy data platforms to modern cloud-based architecture

Skills

Databricks

PySpark

Spark SQL

SQL

Delta Lake

Data Warehousing

ETL/ELT

Orchestration

Data governance

Tools

Delta Lake

Spark

Databricks Notebooks

Data Lakehouse

Job description

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark
• Build and optimize data lakehouse architectures (Bronze–Silver–Gold layers)
• Develop data ingestion frameworks from multiple sources (batch & real-time)
• Implement Delta Lake features including schema evolution, ACID transactions, and optimization
• Create and manage Databricks notebooks, workflows, and jobs for production pipelines
• Ensure data quality, governance, and lineage across pipelines
• Optimize Spark jobs for performance (partitioning, caching, tuning)
• Collaborate with cross-functional teams (Data Scientists, Analysts, Architects)
• Support production deployments, monitoring, and troubleshooting of data pipelines
• Contribute to migration of legacy data platforms to modern cloud-based architecture

Requirements

• 5–10 years of experience in Data Engineering
• Minimum 2–4 years of hands‑on Databricks experience
• Proven experience building enterprise‑scale data platforms and pipelines
• Strong experience in Databricks (Workflows, Notebooks, Jobs)
• PySpark / Spark SQL
• SQL & Data Warehousing concepts
• Hands‑on experience with Delta Lake / Lakehouse architecture
• Data pipeline development & orchestration