Data Engineer |Databricks | 12 month agency contract

PERSOL SINGAPORE PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

PERSOL Singapore PTE. LTD. is seeking an experienced Data Engineer to develop and maintain data processing solutions using Databricks and Python, and to design real-time streaming pipelines with Apache Spark Structured Streaming.

You will troubleshoot and optimize streaming jobs, work with large data volumes, and collaborate with data engineers and stakeholders to deliver effective solutions while following best practices for performance and maintainability.

Qualifications

• 5–10 years of overall experience in data engineering / data development.
• Strong hands-on experience with Databricks.
• Strong experience with Apache Spark Structured Streaming.
• Strong Python programming skills with a good understanding of data processing concepts.

Responsibilities

• Develop and maintain data processing solutions using Databricks and Python.
• Design and implement real-time data processing pipelines using Apache Spark Structured Streaming.
• Develop efficient and scalable data processing logic using strong Python programming fundamentals and data structures.
• Troubleshoot and optimize streaming jobs and data processing workflows.
• Work with large volumes of data and ensure reliable, scalable, and efficient processing.
• Monitor and resolve issues related to streaming pipelines and Databricks workloads.
• Collaborate with data engineers, technical teams, and stakeholders to understand requirements and deliver effective solutions.
• Follow best practices for code quality, performance, maintainability, and operational support.
• Participate in troubleshooting, debugging, and performance optimization of data pipelines.

Skills

Databricks

Apache Spark Structured Streaming

Python programming

Data pipelines

Troubleshooting

Independent & collaborative work

Tools

Databricks

Apache Spark

Job description

Key Responsibilities:

• Develop and maintain data processing solutions using Databricks and Python.
• Design and implement real-time data processing pipelines using Apache Spark Structured Streaming.
• Develop efficient and scalable data processing logic using strong Python programming fundamentals and data structures.
• Troubleshoot and optimize streaming jobs and data processing workflows.
• Work with large volumes of data and ensure reliable, scalable, and efficient processing.
• Monitor and resolve issues related to streaming pipelines and Databricks workloads.
• Collaborate with data engineers, technical teams,and stakeholders to understand requirements and deliver effective solutions.
• Follow best practices for code quality,performance, maintainability, and operational support.
• Participate in troubleshooting, debugging, and performance optimization of data pipelines.

Required Skills & Experience:

• 5–10 years of overall experience in data engineering / data development.
• Strong hands-on experience with Databricks.
• Strong experience with ApacheSpark Structured Streaming.
• Strong Pythonprogramming skills with a good understanding of:
• Experience developing and troubleshooting batch and/or streaming data pipelines.
• Good understanding of data processing concepts anddistributed computing.
• Strong problem-solving and analytical skills.
• Ability to work independently as well as collaboratively with technical teams.

We regret that only shortlisted candidates will be notified.