Senior Big Data Platform Engineer

HCLTech; Kuala Lumpur; On-site; MYR 150,000 - 230,000; Full time

Job summary

HCLTech is seeking a Big Data Platform Engineer to design, build, and maintain scalable data pipelines using Python and Apache Spark in a Linux-based environment. The role focuses on ETL/ELT processes, complex SQL transformations, and ensuring data quality for enterprise analytics.

You will work with Airflow for orchestration, Kubernetes for deployment, and MinIO/S3 storage, while collaborating with DevOps to implement CI/CD pipelines and runbooks.

Qualifications

• Strong programming experience in Python and data engineering
• Hands-on Spark (PySpark) and SQL development and optimization
• Experience with Airflow for workflow orchestration and scheduling
• Kubernetes orchestration and deployment for containerized workloads
• Experience with object storage (MinIO/S3) and ETL/ELT processing
• Performance tuning of data pipelines and Spark jobs
• Experience with Git and version control practices
• Familiarity with DevOps concepts, CI/CD pipelines, and agile delivery

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and Spark
• Implement ETL/ELT processes for large datasets
• Develop and optimize complex SQL queries, data models, and transformations
• Ensure data quality and reliability across platforms
• Operate in Linux environments for deployment, troubleshooting, and tuning
• Create shell scripts for automation and operational tasks
• Monitor and optimize Spark jobs for performance and scalability
• Collaborate with DevOps to improve platform reliability and deployment processes
• Engage with Data Architects and stakeholders to ensure alignment
• Document technical runbooks and guidelines

Skills

Python

Apache Spark

SQL

Apache Airflow

Kubernetes

MinIO/S3

ETL/ELT

Performance Tuning

Git

Agile/Scrum

Tools

Docker

Azure/AWS

Job description

HCLTech is seeking a Big Data Platform Engineer to design, build, and maintain scalable data pipelines using Python and Apache Spark in a Linux-based environment. The role focuses on ETL/ELT processes, complex SQL transformations, and ensuring data quality for enterprise analytics.

You will work with Airflow for orchestration, Kubernetes for deployment, and MinIO/S3 storage, while collaborating with DevOps to implement CI/CD pipelines and runbooks.