Data Engineer (PySpark)

Black Pearl Consult; Abu Dhabi; On-site; AED 120,000 - 200,000; Full time

Job summary

A technology consulting firm in the UAE seeks an experienced Data Engineer (PySpark) to design and maintain scalable data pipelines for production environments. The role demands strong skills in Python, Apache Spark, and proven experience in deploying and tuning data solutions. Join a dynamic team offering a competitive salary and opportunities to work with modern data engineering practices in a collaborative, Dubai-based environment.

Benefits

Competitive salary package

Opportunity to work on production-scale data platforms

Exposure to modern data engineering tools and practices

Qualifications

• Strong proficiency in Python.
• Extensive hands-on experience with Apache Spark (PySpark).
• Experience working with Jupyter Notebooks.
• Strong knowledge of SQL and NoSQL databases.
• Proven experience with Git for version control and CI/CD.
• Hands-on experience with Apache Airflow and/or Jenkins for scheduling and automation.
• Solid understanding of data engineering best practices in production environments.

Responsibilities

• Design, develop, and maintain robust data pipelines using Python and PySpark.
• Perform data ingestion, transformation, cleansing, and validation across datasets.
• Conduct Exploratory Data Analysis (EDA) to identify data patterns and quality issues.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Automate workflows using Apache Airflow and/or Jenkins.

Skills

Python

Apache Spark (PySpark)

SQL databases

NoSQL databases

Git

Apache Airflow

Jupyter Notebooks

Tools

Apache Airflow

Jenkins

Job description

We are seeking an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. The role requires strong hands‑on experience in big data processing, pipeline optimization, and deployment using modern data engineering tools and frameworks.

Key Responsibilities

• Design, develop, and maintain robust, scalable data pipelines using Python and PySpark
• Perform data ingestion, transformation, cleansing, and validation across structured and unstructured datasets
• Conduct Exploratory Data Analysis (EDA) to identify data patterns, anomalies, and quality issues
• Apply data imputation techniques, data linking, and cleansing to ensure high data quality
• Implement feature engineering pipelines to support analytics and downstream use cases
• Optimize Spark jobs for performance, scalability, and cost efficiency
• Deploy and tune production‑grade data pipelines, ensuring reliability and performance
• Automate workflows using Apache Airflow and/or Jenkins
• Collaborate with cross‑functional teams to integrate data solutions into production systems
• Write and maintain unit tests to ensure code quality and reliability
• Manage source code, CI/CD, and deployments using Git, GitHub, and GitHub Actions

Requirements

To be considered for this role, you need to meet the following criteria:

Required Technical Skills

• Strong proficiency in Python
• Extensive hands‑on experience with Apache Spark (PySpark)
• Experience working with Jupyter Notebooks
• Strong knowledge of SQL and NoSQL databases
• Proven experience with Git for version control and CI/CD
• Hands‑on experience with Apache Airflow and/or Jenkins for scheduling and automation
• Solid understanding of data engineering best practices in production environments
• Demonstrated experience in Spark performance tuning and optimization
• Ability to write clean, testable, and maintainable Python code

Mandatory Requirement

• Previous production experience is a MUST, specifically in deploying, tuning, and maintaining data pipelines in production environments

Preferred Qualifications

• Experience working in high‑volume or big data environments
• Strong problem‑solving and analytical skills
• Ability to work independently in a fast‑paced environment

Why Join?

• Competitive salary package
• Opportunity to work on production‑scale data platforms
• Exposure to modern data engineering tools and practices
• Dubai‑based role with a dynamic and collaborative work environment

To view other requirements we have, please visit our website - www.blackpearlconsult.com