Senior Big Data Platform Engineer - Spark, Python & DevOps

HCL Technologies Limited; Kuala Lumpur; On-site; MYR 180,000 - 300,000; Full time

Job summary

HCL Technologies Limited is seeking a highly motivated Big Data Platform Engineer to design, develop, and maintain scalable data processing platforms and pipelines in Kuala Lumpur.

You will apply Python, Apache Spark, SQL, and Linux-based tooling to build reliable, high-performance data solutions for enterprise analytics, while collaborating with Data Architects, Product Owners, and DevOps teams.

Qualifications

• Strong Python programming skills and experience developing data pipelines.
• Hands-on Spark (PySpark) experience and SQL optimization.
• Experience with Linux-based environments and DevOps practices.
• Proficiency with Apache Airflow for workflow orchestration.
• Container orchestration with Kubernetes (AKS/SKE) and cloud storage (MinIO/S3).

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and Apache Spark.
• Implement ETL/ELT processes for large-scale datasets.
• Develop and optimize SQL queries and data models.
• Ensure data quality, integrity, and reliability across the platform.

Skills

Python

Apache Spark (PySpark)

SQL

Linux

DevOps practices

Airflow

Kubernetes

MinIO / S3 Compatible

ETL/ELT

Git

Agile/Scrum

CI/CD pipelines

Shell scripting

Docker

Cloud platforms (Azure/AWS)

Distributed computing

Tools

Apache Spark

Azure/AWS

Docker

Kubernetes

Airflow

Git

MinIO / S3

AKS/SKE

Job description

HCL Technologies Limited is seeking a highly motivated Big Data Platform Engineer to design, develop, and maintain scalable data processing platforms and pipelines in Kuala Lumpur.

You will apply Python, Apache Spark, SQL, and Linux-based tooling to build reliable, high-performance data solutions for enterprise analytics, while collaborating with Data Architects, Product Owners, and DevOps teams.