Lead Data Engineer

Weekday

Delhi

On-site

INR 400,000 - 600,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Weekday is seeking a Lead Data Engineer in Delhi NCR to design and optimize large-scale data pipelines supporting both batch and real-time analytics. You will collaborate with data scientists, product managers, and infrastructure teams to ensure data quality, scalability, and timely insights.

You will lead architecture choices and mentor engineers in a fast-paced environment. The role requires 6+ years in data engineering, hands-on proficiency with Hadoop, Spark, Flink, Kafka, and cloud data

Qualifications

  • 6+ years of data engineering work across batch and real-time systems.
  • Strong knowledge of Hadoop, Spark, and Flink.
  • Experience with Kafka, Pulsar, or RabbitMQ.
  • Proficient in Python, Scala, or Java and SQL over large datasets.
  • Hands-on with cloud data services (AWS/GCP/Azure).
  • Familiar with Docker, Kubernetes, and CI/CD for data workloads.
  • Excellent communication and leadership abilities.

Responsibilities

  • Design and implement end-to-end batch data pipelines using Spark, Hadoop, and Airflow.
  • Develop real-time streaming pipelines with Kafka, Flink, or Spark for low-latency analytics.
  • Optimize pipeline performance through partitioning and data formats like Parquet/ORC.
  • Establish data governance, quality checks, and automated monitoring.
  • Lead capacity planning and scaling for data infrastructure and cost optimization.
  • Mentor junior data engineers and foster collaboration.
  • Collaborate with Data Scientists to translate requirements into scalable solutions.
  • Drive integration of pipelines with dashboards and reporting tools.
  • Stay current with emerging data engineering technologies.
  • Participate in incident response and root cause analysis.

Skills

Distributed computing
Batch processing
Real-time processing
Python/Scala/Java
SQL on big data
Kafka / Pulsar
Docker / Kubernetes

Tools

Hadoop
Spark
Flink
Airflow
Kafka
BigQuery
Redshift
Snowflake

Job description

𝗧𝗵𝗶𝘀 𝗿𝗼𝗹𝗲 𝗶𝘀 𝗳𝗼𝗿 𝗼𝗻𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗪𝗲𝗲𝗸𝗱𝗮𝘆'𝘀 𝗰𝗹𝗶𝗲𝗻𝘁𝘀

𝗦𝗮𝗹𝗮𝗿𝘆 𝗿𝗮𝗻𝗴𑃬: 𝗥𝘀 𝟰𝟬𝟬𝟬𝟬𝟬𝟬 - 𝗥𝘀 𝟲𝟬𝟬𝟬𝟬𝟬𝟬 (𝗶𝗲 𝗜𝗡𝗥 𝟰𝟬-𝟲𝟬 𝗟𝗣𝗔)

Experience: 6+ yrs

Location: NCR, Delhi, India

As a Lead Data Engineer, you will spearhead the design, development, and optimization of large-scale data pipelines that support both batch and real‑time data processing initiatives across the organization. You will collaborate closely with data scientists, product managers, and infrastructure teams to ensure data integrity, scalability, and high performance at every stage of the pipeline, while driving continuous improvement through automation, monitoring, and best practices. Your leadership will shape the data architecture, protect data quality, and enable timely, actionable insights that inform business decisions. In a fast‑moving environment, you will balance technical excellence with strategic vision, fostering a culture of collaboration and innovation within the data engineering team.

Key Responsibilities
  • Design and implement robust, end‑to‑end batch data pipelines using tools such as Apache Spark, Hadoop, and Airflow.
  • Develop real‑time streaming pipelines with technologies like Apache Kafka, Flink, or Spark Structured Streaming to deliver low‑latency analytics.
  • Optimize pipeline performance through partitioning, indexing, and efficient data format selection (e.g., Parquet, ORC).
  • Establish data governance, quality checks, and automated monitoring to ensure data reliability and compliance.
  • Lead capacity planning and scaling strategies for data infrastructure, including cluster management and cost optimization.
  • Mentor and coach junior data engineers, fostering a collaborative and growth‑oriented team culture.
  • Collaborate with Data Scientists to translate analytical requirements into scalable pipeline solutions.
  • Drive integration of data pipelines with downstream applications, dashboards, and reporting tools.
  • Stay current with emerging data engineering technologies and evaluate their applicability to business needs.
  • Participate in incident response and root cause analysis to quickly resolve data pipeline failures.
What Makes You a Great Fit
  • 6+ years of experience in data engineering with a proven track record in both batch and real‑time processing.
  • Deep knowledge of distributed computing frameworks such as Hadoop, Spark, and Flink.
  • Proficiency in message queuing and stream processing systems like Kafka, Pulsar, or RabbitMQ.
  • Strong scripting skills in Python, Scala, or Java, and experience with SQL on large datasets.
  • Hands‑on experience with cloud data services (AWS Glue, Redshift, BigQuery, Snowflake, or Azure Data Factory).
  • Familiarity with containerization (Docker, Kubernetes) and CI/CD pipelines for data workloads.
  • Demonstrated ability to architect and scale data pipelines to handle petabyte‑scale volumes.
  • Excellent communication skills, capable of translating technical concepts for non‑technical stakeholders.
  • A proactive problem‑solver with a passion for data quality, governance, and automation.
  • Leadership aptitude, with experience building, mentoring, and growing data engineering teams.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer
Lead Data Engineer

SourcingXPress • Bengaluru

On-site
INR 1,500,000 - 2,500,000
Lead Data Engineer
Lead Data Engineer

Recro • Bengaluru

On-site
INR 2,500,000 - 4,200,000
Lead Data Engineer – Streaming Data
Lead Data Engineer – Streaming Data

Navikenz • Bengaluru

On-site
INR 1,800,000 - 3,200,000
Lead Data Engineer
Lead Data Engineer

Forward Eye Technologies • Dadri

On-site
INR 1,800,000 - 3,200,000
Lead Data Engineer
Lead Data Engineer

Forward Eye Technologies • Pune District

On-site
INR 1,500,000 - 2,500,000
Lead Data Engineer
Lead Data Engineer

Navikenz • Bengaluru

On-site
INR 1,200,000 - 1,800,000
Lead Data Engineer (Lead Software Engineer)
Lead Data Engineer (Lead Software Engineer)

DigitalXNode • Ahmedabad District

On-site
INR 1,800,000 - 2,600,000
Lead Data Engineer
Lead Data Engineer

Ample Success HR • Pune District

On-site
INR 2,500,000 - 4,500,000
Lead Data Engineer
Lead Data Engineer

Proclink • Pune District

On-site
INR 4,500,000 - 7,000,000
Lead Data Engineer
Lead Data Engineer

Stryker India Private Limited • Gurugram District

Hybrid
INR 3,000,000 - 6,000,000