Databricks Engineer - 12 month agency Contract

PERSOL SINGAPORE PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

PERSOL SINGAPORE PTE. LTD. is seeking an experienced Data Engineer to design, develop, and maintain scalable data solutions for data generation, collection, processing, and analytics in a centralized Singapore location.

You will build production-grade data pipelines using Databricks, PySpark, Python, and SQL, implement ETL/ELT processes, ensure data quality, and collaborate with cross-functional teams to deliver dashboards and data insights.

Qualifications

• Bachelor's degree in CS/IT/Data Science/Engineering or related field.
• At least 4 years of hands-on Data Engineering experience.
• Production-grade Databricks & PySpark experience required.
• Strong SQL for data extraction, transformation, validation, and analysis.

Responsibilities

• Design, develop, and maintain scalable data solutions for data generation, collection, processing, and analytics.
• Build production-grade data pipelines using Databricks, PySpark, Python, and SQL.
• Implement ETL/ELT data movement, data quality checks, and data validation.

Skills

Databricks

PySpark

SQL

Python

ETL/ELT

Data quality

Dashboarding

Education

Bachelor's Degree in Computer Science, Information Technology, Data Science, Engineering

Tools

Git

Power BI

Tableau

Job description

Duration: 1 year

Working Hours: 8.30am - 6pm, Monday - Fridays. Standby and shift work may be required

Location: Central

Responsibilities

• Design, develop, and maintain scalable data solutions for data generation, collection, processing, and analytics.
• Develop and maintain production-grade data pipelines using Databricks, PySpark, Python, and SQL.
• Perform data migration, data transformation, data processing, and integration across multiple systems.
• Build and optimize ETL/ELT pipelines to ensure reliable and efficient data movement across systems.
• Query, clean, transform, and validate large datasets using SQL and PySpark on Databricks.
• Construct complex SQL statements including joins, subqueries, CTEs, aggregations, window functions, and data transformation logic.
• Implement data quality checks, validation rules, and reconciliation processes to ensure data accuracy and consistency.
• Ensure data quality and consistency across systems in accordance with government Instruction Manual (IM8) standards.
• Monitor, troubleshoot, and optimize data pipelines for performance, scalability, and reliability.
• Identify and resolve data pipeline failures, performance bottlenecks, and data quality issues.
• Develop innovative data solutions to improve data accessibility, usability, and operational efficiency.
• Collaborate with data engineers, analysts, application teams, and business stakeholders to understand requirements and translate them into technical solutions.
• Participate in data migration and integration activities across source and target systems.
• Build, enhance, and maintain dashboards for data monitoring, performance reporting, and user behaviour analytics
• Explore and prototype automation solutions to streamline repetitive or manual data workflows.
• Participate in system integration testing (SIT), user acceptance testing (UAT), QA activities, and production deployment.
• Develop and maintain technical documentation, including data mappings, technical specifications, pipeline documentation, and test cases.
• Create user stories and acceptance criteria, manage JIRA tickets, and participate in Agile/Scrum ceremonies and sprint activities.
• Work closely with cross-functional teams to investigate and resolve data-related issues.
• Stay updated with emerging trends and best practices in Databricks, Spark, data engineering, and cloud data platforms.

Requirements

• Bachelor's Degree in Computer Science, Information Technology, Data Science, Engineering, or a related discipline.
• At least 4 years of relevant hands-on Data Engineering experience, with strong practical experience in developing and maintaining data pipelines.
• Must have real industry/production-grade experience with Databricks and PySpark, such as developing, deploying, and supporting live production data solutions.
• Strong proficiency in SQL, with the ability to independently construct and optimize SQL statements for data extraction, transformation, validation, and analysis.
• Strong hands-on experience with PySpark for large-scale/distributed data processing.
• Hands-on experience developing ETL/ELT data pipelines and performing data migration/transformation.
• Strong understanding of Apache Spark architecture and distributed data processing concepts.
• Proficiency in Python for data processing, automation, and scripting.
• Experience working with Databricks notebooks, jobs/workflows, clusters, and data processing environments.
• Experience troubleshooting and optimizing production data pipelines.
• Understanding of data quality, data validation, reconciliation, and monitoring practices.
• Experience with Git or other version control systems.
• Experience with SIT/UAT, system testing, QA, and production deployment.
• Hands-on experience with Power BI, Tableau, or equivalent dashboard/visualisation tools.
• Good understanding of Agile development methodologies, including JIRA, user stories, acceptance criteria, and sprint management.

We regret that only shortlisted applicants would be notified.