Senior Data Engineer - PySpark, Python & CDP Expert

GSSTech Group; Dubai; On-site; AED 360,000 - 520,000; Full time

Job summary

GSSTech Group is seeking a Senior Data Engineer to design and optimize large-scale data pipelines using PySpark, Python, and Cloudera Data Platform (CDP). You will collaborate with data architects, scientists, and stakeholders to deliver scalable data solutions for BI, analytics, and ML initiatives.

The ideal candidate has 8-10+ years in data engineering, strong SQL, data modeling, and experience with CDP and Hadoop ecosystems.

Qualifications

• 8-10+ years of Data Engineering experience.
• Proficient in Python and PySpark.
• Extensive CDP experience and big data orchestration.
• Strong SQL and data modeling abilities.

Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Python.
• Build and optimize batch and distributed data processing with CDP.
• Develop and optimize ETL/ELT pipelines for structured and unstructured data.
• Design scalable data ingestion frameworks from multiple enterprise sources.
• Ensure data quality, governance, and availability across platforms.
• Perform profiling, cleansing, transformation, and validation of data.
• Optimize Spark jobs for performance and resource use.
• Collaborate with Data Scientists on analytics-ready datasets.
• Coordinate with Product Owners and engineers in Agile ceremonies.
• Troubleshoot production data pipelines and resolve issues.
• Contribute to code reviews and engineering best practices.
• Create technical docs and data engineering standards.
• Support continuous improvement of data platforms and processes.

Skills

PySpark

Python

SQL

Data modeling

Data quality

Distributed processing

Agile/Scrum

Tools

CDP (Cloudera Data Platform)

Hadoop

Git

Kubernetes

Job description

GSSTech Group is seeking a Senior Data Engineer to design and optimize large-scale data pipelines using PySpark, Python, and Cloudera Data Platform (CDP). You will collaborate with data architects, scientists, and stakeholders to deliver scalable data solutions for BI, analytics, and ML initiatives.

The ideal candidate has 8-10+ years in data engineering, strong SQL, data modeling, and experience with CDP and Hadoop ecosystems.