Senior Data Pipeline Engineer – Hadoop & Spark

DBS Bank Ltd; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

DBS Bank Ltd in Singapore is seeking a seasoned Technology professional to design, build, and optimize data pipelines within the Hadoop ecosystem. You will tune Spark/SQL ETL performance, automate batch workflows, and manage storage with HDFS and Parquet.

Proficiency in Shell, Python, and Scala, plus AWS S3 experience, will enable you to drive reliable data processing across the bank’s enterprise environment. Strong Linux/Unix skills are essential.

Qualifications

• Proven experience in enterprise-scale application development.
• Strong Hadoop, HDFS, Hive and Impala expertise.
• Proficiency with Spark (Scala/Java) and SQL.
• Experience with Parquet and data formats.
• ETL performance tuning experience.
• Strong Linux/Unix command-line skills.
• Scripting in Shell, Python and Scala.
• Experience with batch automation using REST APIs.

Responsibilities

• Design, develop, and maintain scalable data pipelines in Hadoop ecosystem.
• Tune SQL and Spark ETL performance to reduce latency.
• Automate batch processes using Shell scripts and REST APIs.
• Manage data storage, ingestion, and retrieval with HDFS and Parquet.
• Develop scripts in Shell, Python, and Scala for workflows and integrations.
• Leverage AWS S3 for cloud-based data management and integration.
• Troubleshoot complex issues in Linux/Unix environments.

Skills

Hadoop

HDFS

Hive

Impala

Spark (Scala/Java)

SQL

Linux/Unix

Shell scripting

Python

Scala

REST APIs

AWS S3

Tools

Parquet

REST APIs

AWS

Job description

DBS Bank Ltd in Singapore is seeking a seasoned Technology professional to design, build, and optimize data pipelines within the Hadoop ecosystem. You will tune Spark/SQL ETL performance, automate batch workflows, and manage storage with HDFS and Parquet.

Proficiency in Shell, Python, and Scala, plus AWS S3 experience, will enable you to drive reliable data processing across the bank’s enterprise environment. Strong Linux/Unix skills are essential.