Data Engineer: Scalable Pipelines & AI (Databricks/AWS)

Economic Development Board; Singapore; On-site; SGD 60,000 - 110,000; Full time

Job summary

Economic Development Board's Singapore Global Network (SGN) is seeking a Data Engineer to build end-to-end data pipelines on Databricks and AWS, enabling analytics and BI across global networks.

You will design medallion data architectures (Bronze-Silver-Gold), implement governance with Unity Catalog, and collaborate with stakeholders to deliver reliable datasets for reporting and predictive analytics in a government context.

Qualifications

• 2-5 years of data engineering experience with data pipelines
• Diploma or degree in CS/Engineering/IT/Data Science
• Proficiency with Databricks platform including workflows, notebooks, and cluster management
• Experience implementing medallion architecture for data lake solutions
• Hands-on experience with Unity Catalog for data governance and cataloguing
• Strong Python and SQL skills for data manipulation and transformation
• Experience with Jupyter notebooks and collaborative data science environments
• Working knowledge of AWS data services (S3, Athena, RDS)
• Understanding of ETL/ELT principles and data pipeline design patterns
• Basic knowledge of machine learning concepts and ML pipeline implementation
• Singaporean or PR

Responsibilities

• Design and implement data pipelines using Databricks workflows and notebooks
• Build and maintain medallion architecture (Bronze, Silver, Gold) for data processing and transformation
• Develop ETL processes to integrate data from various SaaS platforms and internal systems
• Implement data governance practices using Unity Catalog for data discovery, lineage, and access control
• Create and maintain Jupyter notebooks for data exploration and collaborative analytics
• Design scalable data storage solutions using AWS services (S3, RDS, Athena)
• Build data models and prepare datasets optimised for reporting and business intelligence
• Implement ML pipelines for basic predictive analytics and data insights
• Ensure data quality, validation, and monitoring across all data workflows
• Collaborate with stakeholders to understand data requirements and deliver analytics-ready datasets

Skills

Databricks

Python

SQL

AWS

Unity Catalog

ETL/ELT

Data governance

Jupyter notebooks

Data modeling

ML pipelines

Data quality

Data visualization

Education

Diploma or degree in CS/Engineering/IT/Data Science

Tools

Databricks

AWS services

Jupyter notebooks

Unity Catalog

RDS/S3/Athena

Power BI/Tableau

Job description

Economic Development Board's Singapore Global Network (SGN) is seeking a Data Engineer to build end-to-end data pipelines on Databricks and AWS, enabling analytics and BI across global networks.

You will design medallion data architectures (Bronze-Silver-Gold), implement governance with Unity Catalog, and collaborate with stakeholders to deliver reliable datasets for reporting and predictive analytics in a government context.