Lead Hadoop Engineer

Talent Smart Soft Solutions; Bengaluru, Chennai District; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Talent Smart Soft Solutions in Bengaluru seeks a Data Engineering Hadoop Developer with 10+ years of big data experience to design, build, and optimize scalable data pipelines on Hadoop ecosystem technologies.

You will work on batch and streaming processing, data integration, performance tuning, and collaborate with business teams to deliver reliable analytics solutions.

Qualifications

• 10 years of hands-on experience in data engineering with Hadoop/big data.
• Strong knowledge of HDFS, Hive and Spark for large-scale data processing.
• Proficiency in Python or Scala for data pipelines and transformations.
• Strong SQL and data modeling fundamentals for reporting and analytics.

Responsibilities

• Design and develop robust ETL/data pipelines using Hadoop ecosystem tools.
• Build and optimize large-scale data processing jobs using Spark and/or Hive.
• Develop workflows using Airflow.
• Write complex SQL/HiveQL for data transformation and reporting needs.
• Ingest structured and unstructured data from multiple sources (Kafka, Sqoop, APIs, files).
• Ensure data quality, lineage, and governance standards are followed.
• Monitor, troubleshoot, and tune jobs for performance and scalability.
• Create technical documentation and follow coding best practices.

Skills

Hadoop/big data

HDFS

Hive

Spark

Python

Scala

SQL

Data modeling

Linux

Shell scripting

Distributed systems

Agile

Tools

Airflow

Git

Job description

**Job Title:** Data Engineering Hadoop Developer (10-15 Years Experience)

**Job Summary**

Seeking a Hadoop Developer with 10 years of experience in big data engineering to design, build, and optimize scalable data pipelines on Hadoop ecosystem technologies. The role involves batch/stream processing, data integration, performance tuning, and close collaboration with business teams.

**Key Responsibilities**

• 1. Design and develop robust ETL/data pipelines using Hadoop ecosystem tools.
• 2. Build and optimize large-scale data processing jobs using Spark and/or Hive.
• 3. Develop workflows using Airflow.
• 4. Write complex SQL/HiveQL for data transformation and reporting needs.
• 5. Ingest structured and unstructured data from multiple sources (Kafka, Sqoop, APIs, files).
• 6. Ensure data quality, lineage, and governance standards are followed.
• 7. Monitor, troubleshoot, and tune jobs for performance and scalability.
• 8. Create technical documentation and follow coding best practices.

**Required Skills**

• 1. 8 years of hands‑on experience in Hadoop/big data development.
• 2. Strong knowledge of HDFS, Hive, Spark,
• 3. String knowledge in Python/Scala
• 4. Strong SQL and data modelling fundamentals.
• 5. Experience with workflow orchestration tools (Airflow).
• 6. Good understanding of Linux, shell scripting, and distributed systems.
• 8. Familiarity with version control (Git) and Agile delivery practices.