Lead Data Engineer

Hungry Bird It Consulting Services; Pune District, Hyderabad; On-site; INR 15,00,000 - 25,00,000; Full time

Job summary

A leading IT consulting firm in India is seeking a Lead Data Engineer to oversee the design and development of data infrastructure. The ideal candidate has significant experience with cloud technologies, ETL pipelines, and data governance, along with strong leadership skills. This role involves collaboration with cross-functional teams and mentoring junior engineers, with a competitive compensation package available.

Qualifications

• 5+ years of experience in data engineering, with 2+ years in a lead role.
• Strong expertise in Python, PySpark, and SQL.
• Hands-on experience with Databricks and AWS.

Responsibilities

• Lead the design, development, and optimization of ETL pipelines.
• Architect scalable and secure data solutions using Databricks and AWS.
• Implement centralized data governance using Databricks Unity Catalog.

Skills

Python

PySpark

SQL

Data Governance

Data Modeling

Education

Bachelors/Masters in Computer Science or related field

Tools

Databricks

AWS (S3, IAM, Lambda, ECR)

Airflow

FastAPI

Columnar Databases (Rockset, Clickhouse, CrateDB)

Job description

Overview

Hello Candidate,

Greetings from Hungry Bird IT Consulting Services Pvt Ltd.

We are hiring Lead Data Engineer for our client.

Job Title: Lead Data Engineer

Location: Hyderabad
Job Type: Full-Time
Experience: 5+ years (2+ years in a leadership role)

Role Overview

We are looking for a highly skilled and experienced Lead Data Engineer to oversee the design, development, and management of scalable data infrastructure and pipelines. You will work closely with cross-functional teams, lead a team of engineers, and drive best practices in data engineering using modern cloud and big data technologies like Databricks, AWS, Airflow, and FastAPI.

Key Responsibilities

• Lead the design, development, and optimization of robust ETL pipelines and data workflows
• Architect scalable and secure data solutions using Databricks and AWS
• Manage and optimize data storage using AWS S3, following data lake and warehouse best practices
• Implement centralized data governance using Databricks Unity Catalog
• Develop data APIs using FastAPI for internal and external integrations
• Use Airflow for workflow orchestration and automation
• Work with columnar databases like Rockset, Clickhouse, and CrateDB
• Ensure data quality, security, and compliance across all systems
• Collaborate with data scientists and analysts to meet analytical and ML needs
• Mentor and guide junior data engineers, promoting code quality and reusability
• Stay updated with new tools and technologies to enhance data infrastructure
• Contribute to cross-functional projects with technical leadership

Required Qualifications

• Bachelors/Masters in Computer Science, Engineering, or a related field
• 5+ years of experience in data engineering, with 2+ years in a lead role
• Strong expertise in Python, PySpark, and SQL
• Hands-on experience with Databricks, AWS (S3, IAM, Lambda, ECR)
• Experience with Airflow, FastAPI, and columnar databases
• Deep understanding of ETL pipelines, data modeling, and data warehousing
• Familiarity with Git, CI/CD, and Agile methodologies
• Knowledge of data governance and compliance standards

Preferred Qualifications

• Experience with real-time data processing/streaming
• Familiarity with MLOps and machine learning workflows
• Certifications in Databricks and/or AWS
• Exposure to data mesh or data fabric architectures
• Understanding of metadata management and data lineage

Tech Stack

• Languages & Frameworks: Python, PySpark, SQL, FastAPI
• Cloud: AWS (S3, IAM, Lambda, ECR)
• Big Data: Databricks
• Orchestration: Airflow
• Databases: Rockset, Clickhouse, CrateDB
• Governance: Databricks Unity Catalog

(Interested candidates can share their CV with us at or reach us at [e-mail])

PLEASE MENTION THE RELEVANT POSITION IN THE SUBJECT LINE OF THE EMAIL.

Example: KRISHNA, HR MANAGER, 7 YEARS, 20 20DAYS NOTICE.

Application Details

Name:

Position applying for:

Total experience:

Notice period:

Current Salary:

Expected Salary:

Thanks and Regards

Aradhana

[phone]