Data Engineering Architect – Databricks

HMG AMERICA LLC; Bengaluru Urban; On-site; INR 4,000,000 - 7,000,000; Full time

Job summary

HMG AMERICA LLC seeks a Lead Data Engineer to drive the design, development, and implementation of scalable data engineering solutions. You will lead metrics-rich pipelines across batch and streaming processing using Scala, Spark, SQL, and cloud platforms.

You will mentor engineers, conduct code reviews, and ensure data quality, governance, and performance. Strong leadership and communication are essential for delivery on large-scale data initiatives.

Qualifications

• 7–12 years of experience in data engineering.
• Strong hands-on with Scala + Apache Spark.
• Excellent knowledge of SQL and database concepts.
• Experience building ETL/ELT data pipelines.
• Experience with Spark Batch and Spark Streaming.
• Experience with Kafka or other streaming technologies.
• Hands-on with at least one cloud platform – AWS / Azure / GCP.
• Strong understanding of data warehousing, data lakes, and distributed processing.
• Experience with performance tuning of Spark applications.
• Good understanding of CI/CD, Git, and Agile methods.
• Strong problem-solving and communication skills.

Responsibilities

• Lead the design and development of high-volume, scalable data pipelines.
• Develop robust batch and real-time data processing solutions using Spark.
• Work with Scala, Spark, and SQL to build optimized data transformation pipelines.
• Design and implement data solutions on cloud platforms such as AWS/Azure/GCP.
• Develop and maintain pipelines for structured and unstructured data.
• Tune performance of Spark jobs, SQL queries, and pipelines.
• Collaborate with Kafka for real-time data processing.
• Ensure data quality, reliability, security, and governance across platforms.
• Provide technical leadership, code reviews, and mentoring.

Skills

Experience 7-12y

Scala & Spark

SQL

ETL/ELT

Spark Batch

Spark Streaming

Kafka

AWS/Azure/GCP

Data Warehousing

Data Lakes

CI/CD

Git

Agile

Problem Solving

Communication

Tools

Databricks

Delta Lake

Airflow

Docker/Kubernetes

Python/PySpark

Job description

We are looking for an experienced Lead Data Engineer to lead the design, development, and implementation of scalable data engineering solutions. The ideal candidate should have strong hands‑on expertise in Scala, Apache Spark, SQL, and cloud data platforms, along with experience in leading technical teams and delivering large‑scale data pipelines.

Key Responsibilities

• Lead the design and development of high-volume, scalable data pipelines and data processing frameworks.
• Develop robust batch and real‑time data processing solutions using Apache Spark.
• Work extensively with Scala, Spark, and SQL to build and optimize data transformation pipelines.
• Design and implement data solutions on cloud platforms such as AWS, Azure, or GCP.
• Develop and maintain data pipelines for structured and unstructured data.
• Perform performance tuning and optimization of Spark jobs, SQL queries, and data pipelines.
• Work with technologies such as Kafka for real‑time/streaming data processing.
• Ensure data quality, reliability, security, and governance across data platforms.
• Provide technical leadership, conduct code reviews, and mentor junior and mid-level engineers.

Mandatory Skills

• 7–12 years of experience in Data Engineering.
• Strong hands‑on experience with Scala + Apache Spark.
• Excellent knowledge of SQL and database concepts.
• Strong experience building ETL/ELT data pipelines.
• Experience with Spark Batch and Spark Streaming.
• Experience with Kafka or other streaming technologies.
• Hands‑on experience with at least one cloud platform – AWS / Azure / GCP.
• Strong understanding of data warehousing, data lakes, and distributed data processing.
• Experience with performance tuning and optimization of Spark applications.
• Good understanding of CI/CD, Git, and Agile methodologies.
• Strong problem‑solving and communication skills.

Good to Have

• Experience with Databricks / Delta Lake.
• Knowledge of Python/PySpark.
• Experience with Airflow or other workflow orchestration tools.
• Experience with Docker/Kubernetes.
• Knowledge of modern Lakehouse architecture.
• Experience leading a team or mentoring data engineers.