Data Engineering Lead: PySpark & Big Data Pipelines

Focal Consulting

Kuala Lumpur

On-site

MYR 120,000 - 180,000

Full time

9 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

13-Month Salary
Yearly Performance Bonus
Monthly Transport Allowance
Monthly Rental Subsidies

Job summary

Focal Consulting in Kuala Lumpur is seeking a Lead Data Engineer to architect scalable PySpark pipelines and guide a team of data engineers. You will design and optimize large-scale batch and streaming workflows, emphasizing reliability and performance.

You will also shape data lake architectures (raw/staged/curated) and collaborate with analysts and ML engineers to deliver production-grade analytics datasets in a fast-paced environment.

Qualifications

  • Bachelor’s or Master’s Degree in Computer Science, Data Engineering, Information Systems, or a related field.
  • 3+ years of data engineering experience, with at least 1 year in leadership/mentoring.
  • Expert-level PySpark, Python, and advanced SQL for complex data transformation and processing.
  • Hands-on experience with Spark clusters, workflow orchestrators (Airflow, Dagster), and data lake formats (Delta Lake or Iceberg).
  • Strong analytical mindset and ability to balance system design with team coordination.

Responsibilities

  • Architect and oversee large-scale batch and streaming data pipelines using PySpark and distributed data frameworks.
  • Lead, mentor, and guide a team of data engineers, conducting code reviews and driving engineering best practices.
  • Design and optimize data lake layers (raw/staged/curated), managing partitioning, storage formats, and schema design.
  • Diagnose bottlenecks, optimize PySpark job performance (memory, shuffling, caching), ensuring high pipeline reliability.
  • Collaborate with data analysts, ML engineers, and business leaders to deliver production-grade analytics datasets.

Skills

PySpark
Python
SQL
Leadership
Mentoring
Airflow
Dagster

Education

Bachelor’s or Master’s Degree in CS/DS/IS

Tools

Apache Spark clusters
Delta Lake
Iceberg

Job description

Focal Consulting in Kuala Lumpur is seeking a Lead Data Engineer to architect scalable PySpark pipelines and guide a team of data engineers. You will design and optimize large-scale batch and streaming workflows, emphasizing reliability and performance.

You will also shape data lake architectures (raw/staged/curated) and collaborate with analysts and ML engineers to deliver production-grade analytics datasets in a fast-paced environment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer - Scalable Pipelines & Data Lakes
Senior Data Engineer - Scalable Pipelines & Data Lakes

Jdynamic Resources • Kuala Lumpur

On-site
MYR 98,000 - 150,000
13-Month Salary
Yearly Performance Bonus
Monthly Transport Allowance
+1
Data Engineering Tech Lead — Databricks & AWS
Data Engineering Tech Lead — Databricks & AWS

BP PLC • Kuala Lumpur

Hybrid
MYR 180,000 - 300,000
Life and health insurance
Learning opportunities
Senior Data Engineer — Scalable AI & Analytics Pipelines
Senior Data Engineer — Scalable AI & Analytics Pipelines

Faculty Digital PTE LTD • Kuala Lumpur

On-site
MYR 180,000 - 280,000
Data Engineer: Cloud-Native Pipelines & Data Warehousing
Data Engineer: Cloud-Native Pipelines & Data Warehousing

Blue Data Digital SDN BHD • Kuala Lumpur

On-site
MYR 120,000 - 180,000
Senior Data Engineer — Scale Pipelines, Drive Data Decisions
Senior Data Engineer — Scale Pipelines, Drive Data Decisions

Grab • Petaling Jaya

On-site
MYR 368,000 - 613,000
Term Life Insurance
Medical Insurance
GrabFlex
+5
Senior Data Engineer: Scalable Pipelines & Platforms
Senior Data Engineer: Scalable Pipelines & Platforms

techstreet • Kuala Lumpur

On-site
MYR 180,000 - 300,000
Senior Data Engineer: Real-Time Pipelines & Lakehouse
Senior Data Engineer: Real-Time Pipelines & Lakehouse

Agensi Pekerjaan Penta Consultancy Sdn. Bhd • Kuala Lumpur

On-site
MYR 180,000 - 240,000
Senior Data Engineer
Senior Data Engineer

Jbsfamily • Kuala Lumpur

On-site
MYR 180,000 - 230,000
Data Engineer — Build Scalable Pipelines & Data Warehouses
Data Engineer — Build Scalable Pipelines & Data Warehouses

S P Setia Berhad • Shah Alam

On-site
MYR 120,000 - 180,000
Medical
Misc. allowance
Loans
+4
Data Engineer: Build Scalable Pipelines & ML Insights
Data Engineer: Build Scalable Pipelines & ML Insights

Data Flows • Kuala Lumpur

On-site
MYR 163,398 - 245,098