PySpark Lead

OX Consultancy Pte. Ltd.

Singapore

Vor Ort

SGD 90.000 - 140.000

Vollzeit

Vor 7 Tagen
Sei unter den ersten Bewerbenden

Eine vollständige Bewerbung in einer Minute

Maßgeschneiderter Lebenslauf und Anschreiben – versandbereit.

Schaffe es an den ATS-Filtern vorbei

Zusammenfassung

OX Consultancy Pte. Ltd. in Singapore seeks a Senior Data Engineer to build and optimize data pipelines with PySpark 3.5.x and Spark SQL. You will work with Delta Lake, Iceberg and Parquet on S3-like storage, and design robust distributed processing architectures.

Skills in Linux, Airflow, Kubernetes, Git/Azure DevOps are essential, plus AI-assisted tooling (VSCode AI, GitHub Copilot) to boost productivity in financial data engineering.

Qualifikationen

  • Proficiency in PySpark 3.5.x and Spark SQL for large-scale data processing.
  • Experience with Delta Lake, Iceberg and Parquet on cloud/object storage.
  • Strong knowledge of distributed systems and data pipelines.

Kenntnisse

PySpark
Python
Spark internals
Distributed systems
Delta Lake
Iceberg
Parquet
Spark SQL
Trino
Hive Metastore
Linux
vi editor
Airflow
S3-compatible storage
Minio
Kubernetes
Git
Azure DevOps
VSCode AI
GitHub Copilot
AI-assisted debugging
Code optimization
Prompt engineering
LLMS workflow productivity

Tools

Apache Spark
Airflow
Kubernetes
Git
Azure DevOps
MinIO
Trino
Hive Metastore
Parquet
Delta Lake
Iceberg

Jobbeschreibung

Required Technical Skills -
  • PySpark 3.5.x
  • - Python
  • - Apache Spark internals and distributed systems architecture
  • - Delta Lake , Iceberg , Parquet - Query Engines Spark SQL , Trino , Hive Metastore
  • -Linux , vi commands
  • - Workflow orchestration (Airflow)
  • - S3 Compatible Object Store ( like Minio )
  • - Containerized deployment environments (Kubernetes)
  • - Git , Azure ADO
  • - Vscode AI
  • - Github Copilot
  • - AI Assisted Debugging , Code Optimization
  • - Prompt Engineering for software development and Experience building productivity workflow using LLMS
Preferred / Nice-to-Have
  • - Trade surveillance domain experience
  • - Financial services data engineering experience
  • - Time-series analytics
  • - Market data processing
Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Software Engineer (Apache / Pyspark / Python) - Contract Role
Software Engineer (Apache / Pyspark / Python) - Contract Role

QUESS ENGINEERING PTE. LTD. • Singapore

Vor Ort
SGD 120.000 - 180.000
Lead Pyspark Developer
Lead Pyspark Developer

Luxoft • Singapore

Vor Ort
SGD 180.000 - 240.000
Lead Pyspark Developer
Lead Pyspark Developer

luxoft information technology (singapore) pte. ltd. • Singapore

Vor Ort
SGD 180.000 - 240.000
Data Engineer / Senior Data Engineer
Data Engineer / Senior Data Engineer

RAPSYS TECHNOLOGIES PTE. LTD. • Singapore

Vor Ort
SGD 120.000 - 180.000
Senior Data Engineer - #1732
Senior Data Engineer - #1732

JOBSTER PRIVATE LTD. • Singapore

Vor Ort
SGD 120.000 - 180.000
Lead Pyspark Developer
Lead Pyspark Developer

DXC Luxoft • Singapore

Vor Ort
SGD 120.000 - 180.000
Data Engineer
Data Engineer

ARYAN SOLUTIONS PTE. LTD. • Singapore

Vor Ort
SGD 90.000 - 150.000
Senior Data Engineer
Senior Data Engineer

optimum solutions (singapore) pte ltd • Singapore

Vor Ort
SGD 120.000 - 180.000
Junior Data Engineer
Junior Data Engineer

INNOVATIQ TECHNOLOGIES PTE. LTD. • Singapore

Vor Ort
SGD 60.000 - 90.000
Snowflake Architect
Snowflake Architect

ITCAN PTE. LIMITED • Singapore

Vor Ort
SGD 90.000 - 125.000