Hadoop Data Engineer

Incedo Inc.; Hyderabad; On-site; INR 1,200,000 - 1,900,000; Full time

Job summary

Incedo Inc. is seeking a Data Engineer in Hyderabad to design, build, and maintain scalable ETL/ELT data pipelines using Hadoop, Hive, HDFS, and Spark/PySpark.

You will optimize SQL queries, develop Python data transformations, and collaborate with analysts and scientists to ensure data quality and clear data flows across distributed storage systems.

Qualifications

• 3–6 years of hands‑on experience in Data Engineering.
• Strong working knowledge of Hadoop ecosystem (HDFS, YARN, MapReduce concepts).
• Proficiency in Hive for data warehousing and query optimization.
• Solid experience with Spark/PySpark for distributed data processing.
• Strong programming skills in Python.
• Advanced SQL skills — query optimization, joins, window functions, performance tuning.

Responsibilities

• Design, build, and maintain scalable ETL/ELT data pipelines using Hadoop, Hive, HDFS, and Spark/PySpark
• Develop and optimize complex SQL queries and Hive scripts for large-scale data processing
• Write clean, efficient, and reusable Python code for data transformation and automation
• Work with structured and unstructured data across distributed storage systems (HDFS)
• Optimize Spark/PySpark jobs for performance, scalability, and resource efficiency
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements
• Ensure data quality, consistency, and integrity across pipelines
• Troubleshoot and resolve issues related to data pipeline failures, performance bottlenecks, and cluster resource management
• Participate in code reviews and follow best practices for data engineering and version control
• Document technical designs, data flows, and pipeline architecture

Education

Bachelor's or Master's degree in Computer Science, Information Technology, or a related field

Tools

Hadoop ecosystem

Hive

Spark/PySpark

Python

SQL

Job description

• Design, build, and maintain scalable ETL/ELT data pipelines using Hadoop, Hive, HDFS, and Spark/PySpark
• Develop and optimize complex SQL queries and Hive scripts for large-scale data processing
• Write clean, efficient, and reusable Python code for data transformation and automation
• Work with structured and unstructured data across distributed storage systems (HDFS)
• Optimize Spark/PySpark jobs for performance, scalability, and resource efficiency
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements
• Ensure data quality, consistency, and integrity across pipelines
• Troubleshoot and resolve issues related to data pipeline failures, performance bottlenecks, and cluster resource management
• Participate in code reviews and follow best practices for data engineering and version control
• Document technical designs, data flows, and pipeline architecture

Required Skills & Experience

• 3–6 years of hands‑on experience in Data Engineering
• Strong working knowledge of Hadoop ecosystem (HDFS, YARN, MapReduce concepts)
• Proficiency in Hive for data warehousing and query optimization
• Solid experience with Spark/PySpark for distributed data processing
• Strong programming skills in Python
• Advanced SQL skills — query optimization, joins, window functions, performance tuning

Good to Have

• Experience with NoSQL databases (HBase, Cassandra)
• Familiarity with CI/CD pipelines for data engineering workflows

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field