Senior PySpark Data Engineer for CDP ETL Pipelines

GSSTech Group; Dubai; On-site; AED 120,000 - 200,000; Full time

Job summary

A prominent technology firm in Dubai is seeking a skilled Data Engineer (PySpark) to develop and maintain scalable data pipelines using the Cloudera Data Platform. This role demands expertise in data ingestion, transformation, and advanced processing techniques. Ideal candidates have a Bachelor's or Master's degree in Computer Science, with over 3 years of experience in data engineering focused on PySpark. The position emphasizes collaboration with teams and commitment to quality data management.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.
• 3+ years of experience as a Data Engineer, with a strong focus on PySpark and the Cloudera Data Platform.

Responsibilities

• Design, develop, and maintain highly scalable ETL pipelines using PySpark.
• Implement and manage data ingestion processes from various sources.
• Use PySpark to cleanse and transform large datasets.
• Conduct performance tuning of PySpark code and Cloudera components.
• Implement data quality checks throughout the pipeline.
• Automate data workflows using Apache Oozie or Airflow.
• Monitor pipeline performance and troubleshoot issues.

Skills

Advanced proficiency in PySpark

Experience with Cloudera Data Platform

Knowledge of data warehousing concepts

Familiarity with Hadoop and Kafka

Experience with orchestration frameworks

Strong scripting skills in Linux

Education

Bachelor’s or Master’s degree in Computer Science

3+ years of experience as a Data Engineer

Tools

Cloudera Manager

Hive

Impala

HDFS

HBase

Apache Oozie

Airflow

Job description

A prominent technology firm in Dubai is seeking a skilled Data Engineer (PySpark) to develop and maintain scalable data pipelines using the Cloudera Data Platform. This role demands expertise in data ingestion, transformation, and advanced processing techniques. Ideal candidates have a Bachelor's or Master's degree in Computer Science, with over 3 years of experience in data engineering focused on PySpark. The position emphasizes collaboration with teams and commitment to quality data management.