Senior PySpark Tech Lead – ETL & Data Lakes

HCL Technologies Middle East FZLLC Dubai Branch; Dubai; On-site; AED 167,000 - 279,000; Full time

Job summary

HCL Technologies Middle East FZLLC Dubai Branch is seeking a Senior PySpark Developer to design, develop, and maintain large-scale data processing systems using PySpark to ingest, transform, and analyze massive data volumes.

You will build scalable ETL/ELT pipelines, optimize Spark jobs, shape data models, and integrate Spark apps with AWS S3, Azure ADLS, and GCP Dataproc, while applying CI/CD practices and unit testing.

Qualifications

• Extensive PySpark ETL/ELT pipeline experience.
• Strong SQL and data modeling skills.
• Cloud data platform experience (AWS/Azure/GCP).
• Familiarity with Parquet and Delta Lake formats.
• Hands-on CI/CD and data workflow automation.

Responsibilities

• Design and build scalable ETL/ELT pipelines to move and transform data.
• Develop PySpark scripts for large-scale data processing (DataFrames, Datasets, Spark SQL).
• Tune Spark jobs for performance, partitions, caching, and shuffles.
• Model data schemas and storage formats (Parquet, Delta Lake, ORC).
• Integrate Spark with cloud storage (S3/ADLS/Blob) and databases.
• Automate data workflows with CI/CD practices and testing.

Skills

Python

SQL

PySpark

Spark SQL

Hadoop

Hive

AWS

Azure

GCP

Delta Lake

Parquet

Avro

HDFS

Snowflake

S3

Tools

Databricks

Airflow

Git

Job description

HCL Technologies Middle East FZLLC Dubai Branch is seeking a Senior PySpark Developer to design, develop, and maintain large-scale data processing systems using PySpark to ingest, transform, and analyze massive data volumes.

You will build scalable ETL/ELT pipelines, optimize Spark jobs, shape data models, and integrate Spark apps with AWS S3, Azure ADLS, and GCP Dataproc, while applying CI/CD practices and unit testing.