Lead PySpark Engineer – Spark & AI-Assisted Debugging

OX Consultancy Pte. Ltd.

Singapore

On-site

SGD 90,000 - 140,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

OX Consultancy Pte. Ltd. in Singapore seeks a Senior Data Engineer to build and optimize data pipelines with PySpark 3.5.x and Spark SQL. You will work with Delta Lake, Iceberg and Parquet on S3-like storage, and design robust distributed processing architectures.

Skills in Linux, Airflow, Kubernetes, Git/Azure DevOps are essential, plus AI-assisted tooling (VSCode AI, GitHub Copilot) to boost productivity in financial data engineering.

Qualifications

  • Proficiency in PySpark 3.5.x and Spark SQL for large-scale data processing.
  • Experience with Delta Lake, Iceberg and Parquet on cloud/object storage.
  • Strong knowledge of distributed systems and data pipelines.

Skills

PySpark
Python
Spark internals
Distributed systems
Delta Lake
Iceberg
Parquet
Spark SQL
Trino
Hive Metastore
Linux
vi editor
Airflow
S3-compatible storage
Minio
Kubernetes
Git
Azure DevOps
VSCode AI
GitHub Copilot
AI-assisted debugging
Code optimization
Prompt engineering
LLMS workflow productivity

Tools

Apache Spark
Airflow
Kubernetes
Git
Azure DevOps
MinIO
Trino
Hive Metastore
Parquet
Delta Lake
Iceberg

Job description

OX Consultancy Pte. Ltd. in Singapore seeks a Senior Data Engineer to build and optimize data pipelines with PySpark 3.5.x and Spark SQL. You will work with Delta Lake, Iceberg and Parquet on S3-like storage, and design robust distributed processing architectures.

Skills in Linux, Airflow, Kubernetes, Git/Azure DevOps are essential, plus AI-assisted tooling (VSCode AI, GitHub Copilot) to boost productivity in financial data engineering.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer AWS / Databricks / PySpark
Senior Data Engineer AWS / Databricks / PySpark

astek singapore innovation technology pte. ltd. • Singapore

On-site
SGD 120,000 - 190,000
Senior Data Engineer – AWS / Databricks / PySpark
Senior Data Engineer – AWS / Databricks / PySpark

ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD. • Singapore

On-site
SGD 120,000 - 180,000
Senior Data Engineer – AWS / Databricks / PySpark
Senior Data Engineer – AWS / Databricks / PySpark

Astek Group • Singapore

On-site
SGD 120,000 - 180,000
Senior Data Engineer: Real-Time Pipelines & Spark
Senior Data Engineer: Real-Time Pipelines & Spark

hyperscal solutions pte. ltd. • Singapore

On-site
SGD 90,000 - 140,000
Data Engineer: Databricks, PySpark & Delta Lake
Data Engineer: Databricks, PySpark & Delta Lake

Espire Infolabs • Singapore

On-site
SGD 70,000 - 110,000
Senior Data Engineer: AWS/Databricks & PySpark Lead
Senior Data Engineer: AWS/Databricks & PySpark Lead

astek singapore innovation technology pte. ltd. • Singapore

On-site
SGD 120,000 - 190,000
Senior Data Engineer - PySpark, Snowflake
Senior Data Engineer - PySpark, Snowflake

Optimum Solutions Pte Ltd • Singapore

On-site
SGD 120,000 - 180,000
Senior Data Engineer – AWS/Databricks/PySpark Architect
Senior Data Engineer – AWS/Databricks/PySpark Architect

ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD. • Singapore

On-site
SGD 120,000 - 180,000
Senior PySpark Lead for Market Surveillance Platform
Senior PySpark Lead for Market Surveillance Platform

Luxoft • Singapore

On-site
SGD 180,000 - 240,000
Senior Data Platform Engineer for AI & Analytics
Senior Data Platform Engineer for AI & Analytics

Talentreq Partners Pte Ltd • Singapore

On-site
SGD 80,000 - 120,000