Production-Scale PySpark Data Engineer

Black Pearl Consult; Abu Dhabi; Presencial; AED 260.000 - 380.000; Jornada completa

Descripción de la vacante

Black Pearl Consult in Abu Dhabi invites an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. You will develop ingestion, transformation, cleansing, and validation processes using Python and PySpark.

The role requires hands-on Spark performance tuning, Airflow/Jenkins automation, CI/CD with Git, and close collaboration with cross-functional teams to deploy robust data solutions.

Formación

• Hands-on PySpark data pipelines in production environments.
• Strong Python coding with clean, testable, maintainable code.
• Experience with Spark performance tuning and data quality checks.

Responsabilidades

• Design, develop, and maintain robust data pipelines using Python and PySpark.
• Ingest, transform, cleanse, and validate data across datasets.
• Perform Exploratory Data Analysis to identify quality issues.
• Implement feature engineering pipelines for analytics.
• Optimize Spark jobs for performance, scalability, and cost.
• Automate workflows with Airflow and Jenkins.

Conocimientos

Python

PySpark

SQL

NoSQL

Airflow

Git

Jupyter

Spark tuning

Herramientas

GitHub Actions

Jenkins

Descripción del empleo

Black Pearl Consult in Abu Dhabi invites an experienced Data Engineer (PySpark) to design, build, optimize, and maintain scalable data pipelines for production environments. You will develop ingestion, transformation, cleansing, and validation processes using Python and PySpark.

The role requires hands-on Spark performance tuning, Airflow/Jenkins automation, CI/CD with Git, and close collaboration with cross-functional teams to deploy robust data solutions.