Senior Cloudera Spark Developer

Indsafri India Private Limited

South Africa

Remote

ZAR 1,200,000 - 1,800,000

Full time

13 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Indsafri India Private Limited in South Africa seeks a Senior Cloudera Spark Developer to design, develop, and optimize large-scale data pipelines using Spark, PySpark, Scala, and SQL within a Cloudera environment.

You will mentor juniors, review code, and collaborate with data engineers and scientists to deliver robust data solutions on HDFS, Hive/Impala, and Spark on YARN, while improving performance and reducing costs.

Qualifications

  • 7+ years of experience in big data development, with a significant focus on Apache Spark.
  • Proficiency in PySpark and Scala for developing Spark applications.
  • Expertise in SQL for data manipulation and complex querying.

Responsibilities

  • Design, develop, and implement high-performance Spark data processing applications.
  • Optimize Spark jobs on Cloudera for performance and cost.
  • Develop and maintain complex SQL queries for data extraction and analysis.
  • Collaborate with data engineers, data scientists, and stakeholders to translate requirements.
  • Build and manage data pipelines for batch and real-time processing.
  • Perform code reviews, testing, and debugging of Spark applications.
  • Troubleshoot performance bottlenecks in Spark environments.
  • Mentor junior developers and share best practices.

Skills

Spark
PySpark
Scala
SQL
Cloudera
HDFS
Hive
Impala
YARN
ETL/ELT

Tools

Kafka
Flink
NoSQL
Airflow
CI/CD

Job description

We are seeking a highly experienced Senior Cloudera Spark Developer with a strong background in big data technologies. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing solutions using Apache Spark within the Cloudera ecosystem. This role requires deep technical expertise in Spark, PySpark, Scala, and SQL, along with a proven track record of delivering robust and scalable data pipelines.

Key Responsibilities:

  • Design, develop, and implement high-performance data processing applications using Apache Spark (PySpark and Scala).
  • Optimize Spark jobs for efficiency, performance, and cost-effectiveness on the Cloudera platform.
  • Develop and maintain complex SQL queries for data extraction, transformation, and analysis.
  • Collaborate with data engineers, data scientists, and business stakeholders to understand data requirements and translate them into technical solutions.
  • Build and manage data pipelines for batch and real-time data processing.
  • Perform code reviews, testing, and debugging of Spark applications.
  • Troubleshoot and resolve performance bottlenecks and issues in Spark environments.
  • Stay up-to-date with the latest advancements in big data technologies, particularly within the Cloudera ecosystem.
  • Mentor junior developers and contribute to best practices in data development.

Required Skills and Qualifications:

  • 7+ years of experience in big data development, with a significant focus on Apache Spark.
  • Proficiency in PySpark and Scala for developing Spark applications.
  • Expertise in SQL for data manipulation and complex querying.
  • Hands-on experience with the Cloudera ecosystem (e.g., HDFS, Hive, Impala, Spark on YARN).
  • Strong understanding of distributed computing principles and data warehousing concepts.
  • Experience with data modeling, ETL/ELT processes, and data pipeline orchestration.
  • Excellent problem-solving and analytical skills.
  • Strong communication and collaboration abilities.

Preferred Qualifications:

  • Experience with cloud platforms (AWS, Azure, GCP) and their big data services.
  • Familiarity with other big data tools and technologies (e.g., Kafka, Flink, NoSQL databases).
  • Experience with CI/CD pipelines and DevOps practices for big data applications.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Cloudera Hadoop Developer
Senior Cloudera Hadoop Developer

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 2,200,000
Senior Cloudera Data Engineer
Senior Cloudera Data Engineer

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,800,000
Senior Cloudera ETL Developer
Senior Cloudera ETL Developer

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,900,000
Senior Data Engineer: Spark, Databricks & Cloud Pipelines
Senior Data Engineer: Spark, Databricks & Cloud Pipelines

Boardroom Appointments • Sandton

On-site
ZAR 900,000 - 1,500,000
Senior Cloudera Platform Engineer
Senior Cloudera Platform Engineer

Indsafri India Private Limited • South Africa

Remote
ZAR 900,000 - 1,500,000
Senior Spark Engineer - Cloudera Data Platform
Senior Spark Engineer - Cloudera Data Platform

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,800,000
Senior Cloudera Data Engineer — Scalable Pipeline Lead
Senior Cloudera Data Engineer — Scalable Pipeline Lead

Indsafri India Private Limited • South Africa

Remote
ZAR 1,200,000 - 1,800,000
Intermediate Cloudera Platform Engineer
Intermediate Cloudera Platform Engineer

Indsafri India Private Limited • South Africa

Remote
ZAR 600,000 - 1,000,000
Senior Cloudera Administrator
Senior Cloudera Administrator

Indsafri India Private Limited • South Africa

Remote
ZAR 900,000 - 1,500,000
Senior Data Engineer
Senior Data Engineer

Skywaves Rise • South Africa

Remote
ZAR 900,000 - 1,300,000
Competitive salary
Comprehensive benefits
Hybrid/remote work options