Senior Data Engineer

HCLTech; Kuala Lumpur; On-site; MYR 150,000 - 230,000; Full time

Job summary

HCLTech is seeking a Big Data Platform Engineer to design, build, and maintain scalable data pipelines using Python and Apache Spark in a Linux-based environment. The role focuses on ETL/ELT processes, complex SQL transformations, and ensuring data quality for enterprise analytics.

You will work with Airflow for orchestration, Kubernetes for deployment, and MinIO/S3 storage, while collaborating with DevOps to implement CI/CD pipelines and runbooks.

Qualifications

• Strong programming experience in Python and data engineering
• Hands-on Spark (PySpark) and SQL development and optimization
• Experience with Airflow for workflow orchestration and scheduling
• Kubernetes orchestration and deployment for containerized workloads
• Experience with object storage (MinIO/S3) and ETL/ELT processing
• Performance tuning of data pipelines and Spark jobs
• Experience with Git and version control practices
• Familiarity with DevOps concepts, CI/CD pipelines, and agile delivery

Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and Spark
• Implement ETL/ELT processes for large datasets
• Develop and optimize complex SQL queries, data models, and transformations
• Ensure data quality and reliability across platforms
• Operate in Linux environments for deployment, troubleshooting, and tuning
• Create shell scripts for automation and operational tasks
• Monitor and optimize Spark jobs for performance and scalability
• Collaborate with DevOps to improve platform reliability and deployment processes
• Engage with Data Architects and stakeholders to ensure alignment
• Document technical runbooks and guidelines

Skills

Python

Apache Spark

SQL

Apache Airflow

Kubernetes

MinIO/S3

ETL/ELT

Performance Tuning

Git

Agile/Scrum

Tools

Docker

Azure/AWS

Job description

Job Description: Big Data Platform Engineer

Role Summary

We are seeking a highly motivated and skilled Big Data Platform Engineer to design, develop, and maintain scalable data processing platforms and pipelines. The ideal candidate should have strong expertise in Python, Apache Spark, SQL, and experience working in Linux-based environments with exposure to DevOps practices. The role involves building reliable, high-performance data solutions that support enterprise-scale analytics and business-critical applications.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and Apache Spark.


• Implement efficient ETL/ELT processes for large-scale structured and unstructured datasets.


• Develop and optimize complex SQL queries, data models, and transformations.


• Ensure data quality, integrity, and reliability across the platform.

Platform Operations

• Work with Linux-based environments for deployment, troubleshooting, and performance tuning.


• Develop and maintain shell scripts for automation and operational tasks.


• Monitor and optimize Spark jobs for performance, scalability, and resource utilization.

DevOps & Automation

• Implement CI/CD pipelines and deployment automation.


• Participate in infrastructure provisioning, monitoring, and release management activities.


• Collaborate with DevOps teams to improve platform reliability and operational efficiency.


• Work closely with Data Architects, Product Owners, and Business Stakeholders.


• Participate in code reviews and ensure adherence to engineering best practices.


• Create and maintain technical documentation and operational runbooks.

Mandatory Skills

Core Big Data Platform Skills

• Strong programming experience in Python


• Hands-on expertise with Apache Spark (PySpark preferred)


• Strong SQL development and query optimization skills


• Apache Airflow for workflow orchestration and scheduling


• Kubernetes (AKS/SKE) for container orchestration and deployment


• MinIO / S3 Compatible Object Storage


• ETL/ELT Processing


• Performance Tuning


• Version Control (Git)


• Agile/Scrum Delivery Model

Good to Have Skills

• Shell Scripting (Bash/KSH)


• Understanding of DevOps practices and CI/CD pipelines


• Docker and Containerization Concepts


• Cloud Platform Experience (Azure/AWS)

Desired Experience

• 6 to 8 years of experience or 8 to 12 years or 12+ years of experience in Data Engineering or Big Data Platform Development.


• Experience working with enterprise-scale data platforms.


• Experience in distributed computing environments.

Strong analytical and problem-solving skills.