Cloudera Hadoop Engineer (Big Data Engineer)

DIXP; Delhi; On-site; INR 900,000 - 1,400,000; Full time

Job summary

Dixpai is seeking a Cloudera Hadoop Engineer to design, build, and optimize large-scale data pipelines on CDP. The role focuses on Hadoop ecosystem tools including Hive, Impala, Spark, and Airflow, with responsibilities spanning ingestion, processing, and performance tuning across distributed clusters.

The ideal candidate will have 4–7 years of Big Data experience, strong SQL skills, and experience with data governance in a Cloudera environment.

Qualifications

• Bachelor's degree in Computer Science, Engineering, Data Engineering, or related field.

Responsibilities

• Design and develop data pipelines and ETL workflows using Hadoop ecosystem tools.
• Build and optimize large-scale data processing jobs on distributed clusters.
• Process structured and unstructured datasets using Hive, Spark, and Impala.
• Develop and manage workloads on Cloudera Data Platform (CDP).
• Design pipelines for data ingestion from multiple sources including databases, APIs, and files.
• Implement batch and scheduled data workflows using Airflow or similar orchestration tools.
• Ensure data quality, transformation, and efficient storage in distributed systems.
• Tune Hive and Impala queries for performance and scalability.
• Monitor and improve performance of data pipelines.

Skills

Hadoop ecosystem

Hive

Impala

Spark

Airflow

CDP

ETL pipelines

SQL

Education

B.Tech/B.E. in Computer Science, Engineering, Data Engineering, or a related field

Tools

CDP

Airflow

Hive

Impala

HDFS

Hue

Job description

Experience: 4 - 7 yrs

Job Type: Full-Time / Contract

Education:

• UG: B.Tech/B.E. in Computer Science, Engineering, Data Engineering, or a related field

Job Description

Project Role Description: We are looking for a Cloudera Hadoop Engineer with strong experience in the Hadoop ecosystem and Cloudera Data Platform (CDP) to build and manage large-scale data processing pipelines. The candidate will work closely with data architects, administrators, and analytics teams to design, develop, and optimize big data workflows and data ingestion pipelines on distributed clusters. The role requires expertise in Hadoop ecosystem technologies such as Hive, Impala, Spark, Airflow, and HDFS, along with experience working on Cloudera-based big data environments.

Key Responsibilities:

• Design and develop data pipelines and ETL workflows using Hadoop ecosystem tools.
• Build and optimize large-scale data processing jobs on distributed clusters.
• Process structured and unstructured datasets using Hive, Spark, and Impala.

Cloudera Platform Development

• Develop and manage workloads on Cloudera Data Platform (CDP).
• Work with services such as Hive, Impala, HDFS, Airflow, and Hue.
• Optimize queries and workloads running on the Cloudera ecosystem.

Data Ingestion & Processing

• Design pipelines for data ingestion from multiple sources including databases, APIs, and files.
• Implement batch and scheduled data workflows using Airflow or similar orchestration tools.
• Ensure data quality, transformation, and efficient storage in distributed systems.

Performance Optimization

• Tune Hive and Impala queries for performance and scalability.
• Optimize data partitioning, indexing, and storage formats for big data processing.
• Monitor and improve performance of data pipelines.
• Work with data scientists, analysts, and business teams to understand data requirements.
• Integrate big data solutions with analytics platforms and reporting systems.
• Collaborate with Cloudera administrators to ensure cluster efficiency.

Documentation & Best Practices

• Maintain documentation for data pipelines, architecture, and workflows.
• Follow best practices for data governance, security, and performance optimization.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, Data Engineering, or related field.
• 4-7 years of experience in Big Data / Hadoop development.

Required Skills:

• Strong experience with Hadoop ecosystem tools.
• Hands-on experience with Hive, Impala, HDFS, and Spark.
• Experience working with Cloudera Data Platform (CDP).
• Experience building ETL pipelines and data workflows.
• Knowledge of SQL and big data query optimization.
• Experience with workflow orchestration tools such as Airflow.
• Good understanding of distributed data processing concepts.

Preferred Skills:

• Experience with Python, Scala, or Java for big data development.
• Knowledge of data lake architecture and big data design patterns.
• Familiarity with data governance tools like Ranger and Atlas.
• Experience integrating big data platforms with analytics and BI tools.