Data Engineer (PySpark)

Black Pearl Consult; Abu Dhabi; On-site; AED 260,000 - 380,000; Full time

Job summary

Black Pearl Consult in Abu Dhabi invites an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. You will develop ingestion, transformation, cleansing, and validation processes using Python and PySpark.

The role requires hands-on Spark performance tuning, Airflow/Jenkins automation, CI/CD with Git, and close collaboration with cross-functional teams to deploy robust data solutions.

Qualifications

• Hands-on PySpark data pipelines in production environments.
• Strong Python coding with clean, testable, maintainable code.
• Experience with Spark performance tuning and data quality checks.

Responsibilities

• Design, develop, and maintain robust data pipelines using Python and PySpark.
• Ingest, transform, cleanse, and validate data across datasets.
• Perform Exploratory Data Analysis to identify quality issues.
• Implement feature engineering pipelines for analytics.
• Optimize Spark jobs for performance, scalability, and cost.
• Automate workflows with Airflow and Jenkins.

Skills

Python

PySpark

SQL

NoSQL

Airflow

Git

Jupyter

Spark tuning

Tools

GitHub Actions

Jenkins

Job description

Abu Dhabi, United Arab Emirates | Posted on 01/30/2026

We are seeking an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. The role requires strong hands-on experience in big data processing, pipeline optimization, and deployment using modern data engineering tools and frameworks.

Key Responsibilities

Design, develop, and maintain robust, scalable data pipelines using Python and PySpark

Perform data ingestion, transformation, cleansing, and validation across structured and unstructured datasets

Conduct Exploratory Data Analysis (EDA) to identify data patterns, anomalies, and quality issues

Apply data imputation techniques, data linking, and cleansing to ensure high data quality

Implement feature engineering pipelines to support analytics and downstream use cases

Optimize Spark jobs for performance, scalability, and cost efficiency

Deploy and tune production-grade data pipelines, ensuring reliability and performance

Automate workflows using Apache Airflow and/or Jenkins

Collaborate with cross-functional teams to integrate data solutions into production systems

Write and maintain unit tests to ensure code quality and reliability

Manage source code, CI/CD, and deployments using Git, GitHub, and GitHub Actions

Requirements

To be considered for this role, you need to meet the following criteria:

Required Technical Skills

Strong proficiency in Python

Extensive hands-on experience with Apache Spark (PySpark)

Experience working with Jupyter Notebooks

Strong knowledge of SQL and NoSQL databases

Proven experience with Git for version control and CI/CD

Hands-on experience with Apache Airflow and/or Jenkins for scheduling and automation

Solid understanding of data engineering best practices in production environments

Demonstrated experience in Spark performance tuning and optimization

Ability to write clean, testable, and maintainable Python code

Mandatory Requirement

Previous production experience is a MUST, specifically in deploying, tuning, and maintaining data pipelines in production environments

Preferred Qualifications

Experience working in high-volume or big data environments

Strong problem-solving and analytical skills

Ability to work independently in a fast-paced environment

Why Join?

Competitive salary package

Opportunity to work on production-scale data platforms

Exposure to modern data engineering tools and practices

Dubai-based role with a dynamic and collaborative work environment