Senior Technical Lead

HCL Technologies Middle East FZLLC Dubai Branch; Dubai; On-site; AED 167,000 - 279,000; Full time

Job summary

HCL Technologies Middle East FZLLC Dubai Branch is seeking a Senior PySpark Developer to design, develop, and maintain large-scale data processing systems using PySpark to ingest, transform, and analyze massive data volumes.

You will build scalable ETL/ELT pipelines, optimize Spark jobs, shape data models, and integrate Spark apps with AWS S3, Azure ADLS, and GCP Dataproc, while applying CI/CD practices and unit testing.

Qualifications

• Extensive PySpark ETL/ELT pipeline experience.
• Strong SQL and data modeling skills.
• Cloud data platform experience (AWS/Azure/GCP).
• Familiarity with Parquet and Delta Lake formats.
• Hands-on CI/CD and data workflow automation.

Responsibilities

• Design and build scalable ETL/ELT pipelines to move and transform data.
• Develop PySpark scripts for large-scale data processing (DataFrames, Datasets, Spark SQL).
• Tune Spark jobs for performance, partitions, caching, and shuffles.
• Model data schemas and storage formats (Parquet, Delta Lake, ORC).
• Integrate Spark with cloud storage (S3/ADLS/Blob) and databases.
• Automate data workflows with CI/CD practices and testing.

Skills

Python

SQL

PySpark

Spark SQL

Hadoop

Hive

AWS

Azure

GCP

Delta Lake

Parquet

Avro

HDFS

Snowflake

S3

Tools

Databricks

Airflow

Git

Job description

Career Opportunities: Senior Technical Lead (183754)

Requisition ID183754-Posted10/21/2026

A PySpark Developer is responsible for designing, developing, and maintaining large-scale, distributed data processing systems and pipelines. They leverage Apache Spark’s Python API (PySpark) to ingest, transform, and analyze massive volumes of structured and unstructured data.

Core Responsibilities

• Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses.
• Big Data Processing: Write optimized PySpark scripts (using DataFrames, Datasets, and Spark SQL) to process terabytes/petabytes of data.
• Performance Tuning: Troubleshoot, debug, and optimize Spark jobs by managing partitions, caching, broadcast variables, and minimizing data shuffling.
• Data Modeling: Assist in designing data architectures, schemas, and storage formats (e.g., Parquet, Delta Lake, ORC).
• Integration: Connect Spark applications with cloud storage (AWS S3, Azure ADLS, Google Cloud Storage) and databases (NoSQL, relational).
• CI/CD & Automation: Deploy data workflows using orchestration tools and maintain code quality through unit testing.

Technical Skills Required

Category

Skills & Technologies

Core Languages Python (Advanced), SQL (Advanced)

Big Data Frameworks Apache Spark / PySpark, Hadoop, Hive

Cloud Platforms AWS (EMR, Glue), Azure (Databricks, Synapse), or GCP (Dataproc)

Storage & Formats Delta Lake, Parquet, Avro, HDFS, Snowflake, Amazon S3