Lead Data Engineer / Engineering Manager

Gurgaon Hire; Mhalunge; On-site; INR 4,000,000 - 6,500,000; Full time

Job summary

Gurgaon Hire is seeking an experienced Data Engineer Lead to design, develop, and govern scalable Databricks data pipelines. You will build end-to-end ETL/ELT solutions and optimize Spark workloads while mentoring engineers and shaping technical standards.

You will collaborate with Architects, Product Owners, and cross-functional teams, ensuring production readiness and successful releases in an Agile environment.

Qualifications

• 8–15 years of Data Engineering experience.
• 4+ years of hands-on Databricks experience.
• Strong experience with PySpark, Python, SQL, and Spark.
• Experience building enterprise-scale data platforms.
• Experience with ETL/ELT and modern Data Lakehouse architectures.
• Experience with Azure or AWS cloud platforms.
• Strong knowledge of Delta Lake and performance optimization.
• Experience in Agile delivery environments.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build end-to-end ETL/ELT solutions using PySpark, Python, and SQL.
• Develop reliable batch and streaming data pipelines.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Implement Delta Lake best practices and Medallion Architecture.
• Develop reusable frameworks and maintain high engineering standards.
• Refactor legacy ETL solutions into modern Databricks-based architectures.
• Ensure production stability through monitoring, alerting, and troubleshooting.
• Define technical standards and best practices.
• Review solution design and code quality.
• Collaborate with Architects, Product Owners, Business Teams, and Platform Teams.
• Mentor junior engineers and provide technical guidance.
• Drive continuous improvement in engineering practices.
• Own delivery of data engineering initiatives.
• Manage project timelines, sprint planning, and technical risks.
• Communicate project status with stakeholders.
• Ensure production readiness and successful releases.

Skills

Databricks

PySpark

Python

SQL

Spark

Delta Lake

Delta Live Tables

Databricks Workflows

Team leadership

Mentoring

Azure/AWS

Git

CI/CD

Tools

Databricks Workspace

Delta Lake

Databricks Workflows / Jobs

Auto Loader

Structured Streaming

Unity Catalog

Delta Live Tables (DLT)

Azure DevOps / GitHub

Job description

Role & responsibilities

Data Engineering

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build end-to-end ETL/ELT solutions using PySpark, Python, and SQL.
• Develop reliable batch and streaming data pipelines.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Implement Delta Lake best practices and Medallion Architecture.
• Develop reusable frameworks and maintain high engineering standards.
• Refactor legacy ETL solutions into modern Databricks-based architectures.
• Ensure production stability through monitoring, alerting, and troubleshooting.

Technical Leadership

• Define technical standards and best practices.
• Review solution design and code quality.
• Collaborate with Architects, Product Owners, Business Teams, and Platform Teams.
• Mentor junior engineers and provide technical guidance.
• Drive continuous improvement in engineering practices.

Delivery & Stakeholder Management

• Own delivery of data engineering initiatives.
• Manage project timelines, sprint planning, and technical risks.
• Communicate project status with stakeholders.
• Ensure production readiness and successful releases.

Leadership (Preferred)

• Lead and mentor Data Engineering teams.
• Support hiring, onboarding, and capability development.
• Conduct technical reviews and mentor engineers.
• Experience managing engineering teams is highly desirable but not mandatory.

Preferred candidate profile

Databricks

• Databricks Workspace
• Delta Lake
• Databricks Workflows / Jobs
• Auto Loader
• Structured Streaming
• Unity Catalog
• Delta Live Tables (DLT)

Programming

• PySpark
• Python
• SQL
• Spark SQL

Data Engineering

• ETL / ELT
• Data Pipeline Development
• Data Warehousing
• Data Modelling
• Batch & Streaming Pipelines
• Performance Optimization

Cloud

• Azure or AWS (GCP is a plus)
• ADLS / S3 DevOps
• Git
• CI/CD
• Azure DevOps / GitHub

Good to Have

• MLflow
• Terraform
• Photon Engine
• Kafka / Event Hubs
• AI/GenAI exposure
• Data Mesh / Data Fabric

Required Experience

• 815 years of Data Engineering experience.
• 4+ years of hands‑on Databricks experience.
• Strong experience with PySpark, Python, SQL, and Spark.
• Experience building enterprise‑scale data platforms.
• Experience with ETL/ELT and modern Data Lakehouse architectures.
• Experience with Azure or AWS cloud platforms.
• Strong knowledge of Delta Lake and performance optimization.
• Experience in Agile delivery environments.

Preferred

• Team leadership or Engineering Management experience.
• Experience managing multiple projects and stakeholders.
• Experience mentoring technical teams.

Certifications (Mandatory) At least one of the following:

• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional

Preferred Certifications

• Azure Data Engineer Associate
• Azure Solutions Architect
• AWS Certified Data Analytics
• Google Professional Data Engineer

Soft Skills

• Excellent communication skills
• Strong problem-solving ability
• Stakeholder management
• Team collaboration
• Leadership and mentoring
• Ownership mindset
• Client-facing experience
• Ability to work in Agile environments