Senior Cloudera Data Engineer

Indsafri India Private Limited

South Africa

Remote

ZAR 1,200,000 - 1,800,000

Full time

13 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Indsafri India Private Limited seeks a highly experienced Senior Cloudera Data Engineer to design, develop, and optimize complex data pipelines within the Cloudera ecosystem. You will translate business requirements into robust data engineering solutions and collaborate with data scientists and stakeholders.

The role requires deep expertise in CDP components (HDFS, YARN, Spark, Hive, Impala, Kudu) and strong Python (PySpark) and SQL skills.

Qualifications

  • 7+ years of hands-on experience in data engineering with big data technologies.
  • Proven expertise in Cloudera Data Platform (CDP) and components like HDFS, YARN, Spark, Hive, Impala, Kudu.
  • Strong programming skills in Python (PySpark) and SQL; experience in designing, developing, and optimizing ETL/ELT workflows.
  • Solid understanding of data warehousing concepts and data modeling techniques.
  • Experience with distributed computing frameworks like Apache Spark.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration abilities.

Responsibilities

  • Design, build, and maintain scalable and reliable data pipelines using Cloudera CDP (CDH/HDP).
  • Develop and optimize ETL/ELT processes using Spark, PySpark, Hive, and Impala.
  • Write efficient and complex SQL queries for data extraction, transformation, and analysis.
  • Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver impactful solutions.
  • Implement data quality checks and ensure data integrity across various data platforms.
  • Monitor and troubleshoot data pipeline performance, identifying and resolving issues proactively.
  • Contribute to architectural decisions and best practices for big data solutions.
  • Mentor junior data engineers and share expertise within the team.
  • Stay up to date with the latest trends and technologies in big data and cloud data engineering.

Skills

Cloudera CDP
PySpark
SQL
ETL/ELT
Big data
Python
Spark
Data Warehousing
Data Modeling
Data Governance
Security best practices

Tools

Docker
Kubernetes
Kafka
Kinesis

Job description

We are seeking a highly experienced Senior Cloudera Data Engineer to join our dynamic data team. The ideal candidate will have extensive experience in designing, developing, and optimizing complex data pipelines and solutions within the Cloudera ecosystem. This role requires a deep understanding of big data technologies and a proven ability to translate business requirements into robust and scalable data engineering solutions.

Key Responsibilities
  • Design, build, and maintain scalable and reliable data pipelines using Cloudera CDP (CDH/HDP).
  • Develop and optimize ETL/ELT processes using Spark, PySpark, Hive, and Impala.
  • Write efficient and complex SQL queries for data extraction, transformation, and analysis.
  • Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver impactful solutions.
  • Implement data quality checks and ensure data integrity across various data platforms.
  • Monitor and troubleshoot data pipeline performance, identifying and resolving issues proactively.
  • Contribute to architectural decisions and best practices for big data solutions.
  • Mentor junior data engineers and share expertise within the team.
  • Stay up to date with the latest trends and technologies in big data and cloud data engineering.
Required Skills and Qualifications
  • 7+ years of hands-on experience in data engineering, with a strong focus on big data technologies.
  • Proven expertise in the Cloudera Data Platform (CDP) and its components (e.g., HDFS, YARN, Spark, Hive, Impala, Kudu).
  • Strong programming skills in Python (PySpark) and proficiency in SQL.Experience in designing, developing, and optimizing complex ETL/ELT workflows.
  • Solid understanding of data warehousing concepts and data modeling techniques.
  • Experience with distributed computing frameworks like Apache Spark.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration abilities.
Preferred Qualifications
  • Experience with cloud platforms (AWS, Azure, GCP) and their data services.
  • Familiarity with containerization technologies (Docker, Kubernetes).
  • Experience with streaming data technologies (Kafka, Kinesis).
  • Knowledge of data governance and security best practices.
Skills

Docker Python AWS GCP Azure analytical skills SQL PySpark Problem-Solving communication Kubernetes Collaboration Data Warehousing Data Modeling Data Governance Security best practices apache spark hdfs kafka impala spark Hive YARN Kinesis Big Data technologies Data Engineering Cloudera Data Platform (CDP) Kudu ETL/ELT workflows Distributed Computing Frameworks

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Cloudera Platform Engineer
Senior Cloudera Platform Engineer

Indsafri India Private Limited • South Africa

Remote
ZAR 900,000 - 1,500,000
Senior Cloudera Hadoop Developer
Senior Cloudera Hadoop Developer

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 2,200,000
Senior Cloudera Spark Developer
Senior Cloudera Spark Developer

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,800,000
Cloudera Data Architect
Cloudera Data Architect

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,800,000
Senior Cloudera ETL Developer
Senior Cloudera ETL Developer

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,900,000
Senior Cloudera Data Engineer — Scalable Pipeline Lead
Senior Cloudera Data Engineer — Scalable Pipeline Lead

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,800,000
Intermediate Cloudera Platform Engineer
Intermediate Cloudera Platform Engineer

Indsafri India Private Limited • South Africa

Remote
ZAR 600,000 - 1,000,000
Senior Cloudera Administrator
Senior Cloudera Administrator

Indsafri India Private Limited • South Africa

Remote
ZAR 900,000 - 1,500,000
Senior Data Engineer - Cloud Data Architect & Analytics
Senior Data Engineer - Cloud Data Architect & Analytics

Boardroom Appointments • Johannesburg

On-site
ZAR 800,000 - 1,200,000
Senior Data Engineer: Spark, Databricks & Cloud Pipelines
Senior Data Engineer: Spark, Databricks & Cloud Pipelines

Boardroom Appointments • Sandton

On-site
ZAR 900,000 - 1,500,000