Data Engineering Architect ( Databricks)

3minds Esolutions; Hyderabad, Bangalore Rural; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

3minds Esolutions is seeking a Lead Data Engineer in Bangalore to lead the design, development, and delivery of scalable data solutions. You will drive end-to-end data pipelines using Scala, Apache Spark, and SQL, and mentor a technical team across batch and streaming workloads.

You will design data platforms on AWS/Azure/GCP, optimize Spark jobs, and collaborate with cross-functional teams to enable data-driven decision making. Strong communication and agile practices are essential.

Qualifications

• 7–12 years of experience in Data Engineering.
• Strong hands-on experience with Scala + Apache Spark.
• Excellent knowledge of SQL and database concepts.
• Strong experience building ETL/ELT data pipelines.
• Experience with Spark Batch and Spark Streaming.
• Experience with Kafka or other streaming technologies.
• Hands-on experience with at least one cloud platform – AWS / Azure / GCP.
• Strong understanding of data warehousing, data lakes, and distributed data processing.
• Experience with performance tuning and optimization of Spark applications.
• Good understanding of CI/CD, Git, and Agile methodologies.
• Strong problem-solving and communication skills.

Responsibilities

• Lead the design and development of high-volume, scalable data pipelines and data processing frameworks.
• Develop robust batch and real-time data processing solutions using Apache Spark.
• Work extensively with Scala, Spark, and SQL to build and optimize data transformation pipelines.
• Design and implement data solutions on cloud platforms such as AWS, Azure, or GCP.
• Develop and maintain data pipelines for structured and unstructured data.
• Perform performance tuning and optimization of Spark jobs, SQL queries, and data pipelines.
• Work with technologies such as Kafka for real-time/streaming data processing.

Skills

Scala

Apache Spark

SQL

ETL/ELT pipelines

Kafka

Cloud platforms

CI/CD

Git

Agile methodologies

Data warehousing

Tools

AWS

Azure

GCP

Job description

Job Title: Lead Data Engineer
Experience: 712 Years
Location: Bangalore
Employment Type: Full-Time
Work Mode: Work from Office

( Immediate joiner )

Job Summary

We are looking for an experienced Lead Data Engineer to lead the design, development, and implementation of scalable data engineering solutions. The ideal candidate should have strong hands‑on expertise in Scala, Apache Spark, SQL, and cloud data platforms, along with experience in leading technical teams and delivering large‑scale data pipelines.

Key Responsibilities

• Lead the design and development of high‑volume, scalable data pipelines and data processing frameworks.
• Develop robust batch and real‑time data processing solutions using Apache Spark.
• Work extensively with Scala, Spark, and SQL to build and optimize data transformation pipelines.
• Design and implement data solutions on cloud platforms such as AWS, Azure, or GCP.
• Develop and maintain data pipelines for structured and unstructured data.
• Perform performance tuning and optimization of Spark jobs, SQL queries, and data pipelines.
• Work with technologies such as Kafka for real‑time/streaming data processing.

Mandatory Skills

• 7–12 years of experience in Data Engineering.
• Strong hands‑on experience with Scala + Apache Spark.
• Excellent knowledge of SQL and database concepts.
• Strong experience building ETL/ELT data pipelines.
• Experience with Spark Batch and Spark Streaming.
• Experience with Kafka or other streaming technologies.
• Hands‑on experience with at least one cloud platform – AWS / Azure / GCP.
• Strong understanding of data warehousing, data lakes, and distributed data processing.
• Experience with performance tuning and optimization of Spark applications.
• Good understanding of CI/CD, Git, and Agile methodologies.
• Strong problem‑solving and communication skills.