Senior Lead Data Engineer

Jobtailor; Hyderabad; On-site; INR 4,500,000 - 6,000,000; Full time

Job summary

Jobtailor is seeking an senior data engineering lead to architect and guide scalable data platforms in a Hyderabad-based role. You will design end-to-end data engineering solutions, build large-scale pipelines with Spark, Python, and SQL, and manage orchestration with Airflow.

You will drive Databricks-based platforms on AWS, mentor teams, and collaborate with stakeholders to deliver business value while upholding data standards and best practices.

Qualifications

• Bachelor's or Master's degree in Computer Science or related field.
• 9+ years in data engineering, with lead experience.
• Strong SQL, Python, and Spark expertise.
• Hands-on with Apache Airflow and Databricks.
• Experience on AWS data stack (S3, Glue, Redshift, Lambda).
• Proven data architecture, modeling, and big data systems.
• Experience leading engineering teams and projects.

Responsibilities

• Architect, build, and lead scalable data platforms and teams.
• Design end-to-end data engineering solutions.
• Build and optimize large-scale pipelines with Spark, Python, SQL.
• Design and manage orchestration using Apache Airflow.
• Develop on AWS (S3, Glue, Redshift, Lambda) at scale.
• Drive Databricks-based platform development and Lakehouse design.
• Mentor engineers and collaborate with stakeholders to deliver value.
• Implement CI/CD practices including GitHub Actions.
• Ensure platform scalability, performance, and reliability.

Skills

SQL

Python

Apache Spark

Leadership

Communication

Data modeling

Big data

Problem solving

Data architecture

Team mentoring

Education

Bachelor's/Master's in CS

Tools

Airflow

Databricks

DBT

AWS

Delta Lake

S3

Redshift

GitHub Actions

Job description

Responsibilities

• Architect, build, and lead scalable data platforms
• Lead the design and implementation of end-to-end data engineering solutions
• Architect and optimize large-scale data pipelines using Spark, Python, and SQL
• Design and manage orchestration frameworks using Apache Airflow
• Build scalable and cost-efficient solutions on AWS ecosystem (S3, Glue, Redshift, Lambda)
• Drive development and optimization of Databricks-based data platforms
• Mentor and guide data engineering teams
• Collaborate with cross-functional stakeholders to deliver business value
• Implement and drive CI/CD practices, including GitHub Actions
• Ensure data platform scalability, performance, and reliability
• Develop and maintain dimensional models (Star/Snowflake schemas) for analytics and reporting
• Build and optimize data Lakehouse architectures using Databricks & Delta Lake
• Define data standards, naming conventions, and modeling best practices

Requirements

• Bachelor’s/Master’s degree in Computer Science or related discipline
• Typically 9+ years of experience in data engineering (Lead role)
• Strong expertise in SQL, Python, and Apache Spark
• Extensive hands-on experience with Apache Airflow
• Deep experience with AWS cloud services (S3, Glue, Redshift, Lambda)
• Experience building solutions on Databricks platform
• Experience in DBT
• Proven experience in data architecture, modeling, and big data systems
• Experience leading engineering teams and projects
• Strong analytical, problem-solving, and communication skills