Lead Data Engineer

3minds Esolutions; Hyderabad, Bangalore Rural; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

3minds Esolutions in Bangalore is seeking a Lead Data Engineer to drive design and implementation of scalable data pipelines. The role demands hands-on expertise in Scala, Apache Spark, SQL and cloud platforms, with leadership experience on large-scale data initiatives.

The candidate will build and optimize high-volume pipelines, batch and streaming processes, and data transformations using Spark, Scala, and SQL, while collaborating with cloud teams and data stakeholders.

Qualifications

• 7–12 years of experience in Data Engineering.
• Hands-on with Scala and Apache Spark.
• Strong SQL and database concepts.
• Experience with ETL/ELT pipelines.
• Spark Batch and Spark Streaming experience.
• Kafka or other streaming technologies.
• Cloud platform experience: AWS, Azure, or GCP.
• Knowledge of data warehouses and data lakes.
• CI/CD, Git, and Agile methodologies.
• Strong problem-solving and communication skills.

Responsibilities

• Lead the design and development of high-volume, scalable data pipelines and data processing frameworks.
• Develop robust batch and real-time data processing solutions using Apache Spark.
• Work with Scala, Spark, and SQL to build and optimize data transformation pipelines.
• Design and implement data solutions on cloud platforms such as AWS, Azure, or GCP.
• Develop and maintain data pipelines for structured and unstructured data.
• Perform performance tuning and optimization of Spark jobs, SQL queries, and data pipelines.
• Work with Kafka for real-time/streaming data processing.

Skills

Scala

Apache Spark

SQL

ETL/ELT

Spark Streaming

Kafka

AWS

Azure

GCP

Data warehousing

CI/CD

Git

Agile

Communication

Job description

Job Title: Lead Data Engineer
Experience: 712 Years
Location: Bangalore
Employment Type: Full-Time
Work Mode: Work from Office

( Immediate joiner )

Job Summary

We are looking for an experienced Lead Data Engineer to lead the design, development, and implementation of scalable data engineering solutions. The ideal candidate should have strong hands-on expertise in Scala, Apache Spark, SQL, and cloud data platforms, along with experience in leading technical teams and delivering large-scale data pipelines.

Key Responsibilities

• Lead the design and development of high-volume, scalable data pipelines and data processing frameworks.
• Develop robust batch and real-time data processing solutions using Apache Spark.
• Work extensively with Scala, Spark, and SQL to build and optimize data transformation pipelines.
• Design and implement data solutions on cloud platforms such as AWS, Azure, or GCP.
• Develop and maintain data pipelines for structured and unstructured data.
• Perform performance tuning and optimization of Spark jobs, SQL queries, and data pipelines.
• Work with technologies such as Kafka for real-time/streaming data processing.

Mandatory Skills

• 7–12 years of experience in Data Engineering.
• Strong hands-on experience with Scala + Apache Spark.
• Excellent knowledge of SQL and database concepts.
• Strong experience building ETL/ELT data pipelines.
• Experience with Spark Batch and Spark Streaming.
• Experience with Kafka or other streaming technologies.
• Hands-on experience with at least one cloud platform – AWS / Azure / GCP.
• Strong understanding of data warehousing, data lakes, and distributed data processing.
• Experience with performance tuning and optimization of Spark applications.Good understanding of CI/CD, Git, and Agile methodologies.
• Strong problem-solving and communication skills.