Streaming Data Engineer (Databricks & Python)

PERSOL SINGAPORE PTE. LTD.

Singapore

On-site

SGD 120,000 - 180,000

Full time

10 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

PERSOL Singapore Pte Ltd is seeking an experienced data engineer to develop and maintain data processing solutions using Databricks and Python in a fast-paced environment. You will design real-time pipelines with Apache Spark Structured Streaming, troubleshoot streaming jobs, and collaborate with data engineers and stakeholders to deliver scalable, reliable workflows.

The ideal candidate brings 5–10 years of data engineering experience, a solid foundation in distributed computing, and a

Qualifications

5–10 years of overall data engineering / development experience.
Strong hands-on Databricks experience.
Strong Apache Spark Structured Streaming experience.
Proficient Python programming with good data-structures knowledge.
Experience developing and troubleshooting batch and/or streaming data pipelines.

Responsibilities

Develop and maintain data processing solutions using Databricks and Python.
Design and implement real-time data processing pipelines using Spark Structured Streaming.
Develop efficient and scalable data processing logic with Python and data structures.
Troubleshoot and optimize streaming jobs and data processing workflows.
Work with large volumes of data and ensure reliable, scalable processing.
Monitor and resolve issues related to streaming pipelines and Databricks workloads.
Collaborate with data engineers, technical teams, and stakeholders to deliver effective solutions.
Follow best practices for code quality, performance, maintainability, and operational support.
Participate in troubleshooting, debugging, and performance optimization of data pipelines.