CDP PySpark Data Engineer – Scalable Pipelines & Quality

Clover Infotech; Dubai; On-site; AED 240,000 - 360,000; Full time

Job summary

Clover Infotech in the United Arab Emirates seeks a Data Engineer with strong PySpark and Cloudera Data Platform (CDP) expertise to design and maintain scalable data pipelines. You will ensure data quality and availability across the organization and work with a team of data engineers to drive business insights.

The role emphasizes ingestion, transformation, and optimization, leveraging big data ecosystems and cloud-native tools to deliver robust data solutions for analytics and reporting.

Qualifications

• Proven experience building scalable ETL pipelines with PySpark on CDP.
• Strong data ingestion, transformation, and validation capabilities.
• Experience optimizing PySpark and CDP components for performance.
• Familiarity with data quality checks and monitoring.
• Ability to collaborate with data engineers to deliver business insights.

Responsibilities

• Data Pipeline Development: Design, develop, and maintain scalable ETL pipelines using PySpark on CDP.
• Data Ingestion: Manage ingestion from relational DBs, APIs, and files into data lake/warehouse on CDP.
• Data Transformation and Processing: Use PySpark to cleanse and transform large datasets for analytics.
• Performance Optimization: Tune PySpark and CDP for resource efficiency and speed.
• Data Quality and Validation: Implement checks to ensure data accuracy and reliability.
• Automation and Orchestration: Automate data workflows with Oozie or Airflow within CDP.
• Monitoring and Maintenance: Monitor pipelines and troubleshoot issues.

Skills

PySpark

CDP

ETL development

Data pipelines

Big data

Tools

Apache Oozie

Airflow

Job description

Clover Infotech in the United Arab Emirates seeks a Data Engineer with strong PySpark and Cloudera Data Platform (CDP) expertise to design and maintain scalable data pipelines. You will ensure data quality and availability across the organization and work with a team of data engineers to drive business insights.

The role emphasizes ingestion, transformation, and optimization, leveraging big data ecosystems and cloud-native tools to deliver robust data solutions for analytics and reporting.