Technical Lead

Software Company; Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Software Company in Bengaluru, India, is seeking a seasoned Big Data Engineer to develop and maintain scalable data solutions using the Hadoop ecosystem, including HDFS, Hive, Spark, Sqoop, Oozie and YARN. You will build and optimize Spark and Hive transformations and ensure robust ingestion from multiple sources.

You will collaborate with data architects and analysts, implement data quality and reconciliation processes, and tune Spark/Hadoop jobs for performance and resource utilization on

Qualifications

• Proficient in Hadoop ecosystem and data processing
• Experience with HDFS, Hive, Spark, Sqoop, Oozie and YARN
• Strong SQL and HiveQL skills for data transformations
• Hands-on with ETL pipelines and batch processing of big data
• Understanding of data warehousing concepts and data storage best practices
• Familiarity with Python/Scala/Java for data processing
• Knowledge of distributed computing and performance tuning
• Experience building scalable data ingestion from multiple sources
• Ability to implement data quality, validation, and reconciliation processes
• Experience collaborating with data architects, analysts, and apps teams

Responsibilities

• Develop and maintain scalable Big Data solutions using the Hadoop ecosystem.
• Work with HDFS, Hive, Spark, Sqoop, Oozie, and YARN.
• Develop and optimize Hive SQL queries and data transformations.
• Build batch-processing and ETL pipelines using Apache Spark.
• Perform data ingestion from relational databases and external sources using Sqoop or equivalent tools.
• Manage data storage, partitioning, bucketing, compression, and schema evolution.
• Monitor and troubleshoot Hadoop jobs and cluster performance.
• Work with Python / Scala / Java for data processing.
• Implement data quality, validation, and reconciliation processes.
• Collaborate with Data Architects, Analysts, and application teams to deliver data solutions.
• Optimize Spark/Hadoop jobs for performance and resource utilization.

Skills

Hadoop ecosystem

Distributed computing concepts

Performance tuning

Data quality

ETL pipelines

Batch processing

SQL

Big Data ETL

Data Warehousing

Python / Scala / Java

Spark

HDFS

Hive / HiveQL

YARN

Tools

Hadoop

HDFS

Hive

Spark

Sqoop

Oozie

YARN

HiveQL

Python

Scala

Java

Job description

Role & responsibilities

• Develop and maintain scalable Big Data solutions using the Hadoop ecosystem.
• Work with HDFS, Hive, Spark, Sqoop, Oozie, and YARN.
• Develop and optimize Hive SQL queries and data transformations.
• Build batch-processing and ETL pipelines using Apache Spark.
• Perform data ingestion from relational databases and external sources using Sqoop or equivalent tools.
• Manage data storage, partitioning, bucketing, compression, and schema evolution.
• Monitor and troubleshoot Hadoop jobs and cluster performance.
• Work with Python / Scala / Java for data processing.
• Implement data quality, validation, and reconciliation processes.
• Collaborate with Data Architects, Analysts, and application teams to deliver data solutions.
• Optimize Spark/Hadoop jobs for performance and resource utilization.

Preferred candidate profile

• Hadoop
• HDFS
• Hive / HiveQL
• Apache Spark
• YARN
• SQL
• Big Data ETL
• Data Warehousing
• Python / Scala / Java
• Distributed computing concepts
• Performance tuning and troubleshooting