Production-Scale PySpark Data Engineer

Black Pearl Consult; Abu Dhabi; On-site; AED 260,000 - 380,000; Full time

Job summary

Black Pearl Consult in Abu Dhabi invites an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. You will develop ingestion, transformation, cleansing, and validation processes using Python and PySpark.

The role requires hands-on Spark performance tuning, Airflow/Jenkins automation, CI/CD with Git, and close collaboration with cross-functional teams to deploy robust data solutions.

Qualifications

• Hands-on PySpark data pipelines in production environments.
• Strong Python coding with clean, testable, maintainable code.
• Experience with Spark performance tuning and data quality checks.

Responsibilities

• Design, develop, and maintain robust data pipelines using Python and PySpark.
• Ingest, transform, cleanse, and validate data across datasets.
• Perform Exploratory Data Analysis to identify quality issues.
• Implement feature engineering pipelines for analytics.
• Optimize Spark jobs for performance, scalability, and cost.
• Automate workflows with Airflow and Jenkins.

Skills

Python

PySpark

SQL

NoSQL

Airflow

Git

Jupyter

Spark tuning

Tools

GitHub Actions

Jenkins

Job description

Black Pearl Consult in Abu Dhabi invites an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. You will develop ingestion, transformation, cleansing, and validation processes using Python and PySpark.

The role requires hands-on Spark performance tuning, Airflow/Jenkins automation, CI/CD with Git, and close collaboration with cross-functional teams to deploy robust data solutions.