Databricks & Python Engineer| 12 month agency contract

PERSOL SINGAPORE PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

PERSOL Singapore Pte Ltd is seeking an experienced data engineer to develop and maintain data processing solutions using Databricks and Python in a fast-paced environment. You will design real-time pipelines with Apache Spark Structured Streaming, troubleshoot streaming jobs, and collaborate with data engineers and stakeholders to deliver scalable, reliable workflows.

The ideal candidate brings 5–10 years of data engineering experience, a solid foundation in distributed computing, and a

Qualifications

• 5–10 years of overall data engineering / development experience.
• Strong hands-on Databricks experience.
• Strong Apache Spark Structured Streaming experience.
• Proficient Python programming with good data-structures knowledge.
• Experience developing and troubleshooting batch and/or streaming data pipelines.

Responsibilities

• Develop and maintain data processing solutions using Databricks and Python.
• Design and implement real-time data processing pipelines using Spark Structured Streaming.
• Develop efficient and scalable data processing logic with Python and data structures.
• Troubleshoot and optimize streaming jobs and data processing workflows.
• Work with large volumes of data and ensure reliable, scalable processing.
• Monitor and resolve issues related to streaming pipelines and Databricks workloads.
• Collaborate with data engineers, technical teams, and stakeholders to deliver effective solutions.
• Follow best practices for code quality, performance, maintainability, and operational support.
• Participate in troubleshooting, debugging, and performance optimization of data pipelines.

Skills

Databricks

Apache Spark Structured Streaming

Python programming

Data engineering

Problem-solving

Tools

Databricks

Apache Spark

Job description

Key Responsibilities:

• Develop and maintain data processing solutions using Databricks and Python.
• Design and implement real-time data processing pipelines using Apache Spark Structured Streaming.
• Develop efficient and scalable data processing logic using strong Python programming fundamentals and data structures.
• Troubleshoot and optimize streaming jobs and data processing workflows.
• Work with large volumes of data and ensure reliable, scalable, and efficient processing.
• Monitor and resolve issues related to streaming pipelines and Databricks workloads.
• Collaborate with data engineers, technical teams,and stakeholders to understand requirements and deliver effective solutions.
• Follow best practices for code quality,performance, maintainability, and operational support.
• Participate in troubleshooting, debugging, and performance optimization of data pipelines.

Required Skills & Experience:

• 5–10 years of overall experience in data engineering / data development.
• Strong hands-on experience with Databricks.
• Strong experience with ApacheSpark Structured Streaming.
• Strong Pythonprogramming skills with a good understanding of:
• Experience developing and troubleshooting batch and/or streaming data pipelines.
• Good understanding of data processing concepts anddistributed computing.
• Strong problem-solving and analytical skills.
• Ability to work independently as well as collaboratively with technical teams.

We regret that only shortlisted candidates will be notified.

PERSOL Singapore Pte Ltd

EA License No: 01C4394| Saravanan| Reg No: 01C4394