Data Engineer

OPTIMUM SOLUTIONS (SINGAPORE) PTE LTD; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Optimum Solutions (Singapore) Pte Ltd is seeking a data engineer to design and maintain production-grade data pipelines. You will clean and transform data, build ETL processes, and ensure data accuracy across systems using SQL, Python, and cloud services.

You will work with various databases and big data tools, implement scalable data architectures, and collaborate with engineering and analytics teams to bridge the gap between data engineering and business insights.

Qualifications

• Proficient in data cleaning and transformation using SQL, pandas, R.
• Experience building ETL pipelines (SSIS, AWS DMS, Python, AWS Lambda, ECS, EventBridge, AWS Glue).
• Proficient in database design and multiple databases (SQL, PostgreSQL, AWS S3, Athena, MongoDB, MySQL, SQLite).
• Experience with cloud platforms (AWS, Azure, Google Cloud).
• Experience in big data environments and data pipelines (ETL/ELT).
• Knowledge of system design, data structures and algorithms.
• Familiar with data modelling, data access, and storage like Data Lake/Warehouse.
• Familiar with REST APIs and web protocols.
• Familiar with big data frameworks (Hadoop, Spark, Kafka, RabbitMQ).
• Familiar with web scraping tools (Beautiful Soup, CasperJS, PhantomJS, Selenium, Node.js).
• Familiar with data governance and security practices.
• Comfortable in SQL/Python scripting and Windows/Linux environments.
• Interest in bridging engineering and analytics.

Skills

SQL

Python

Data cleaning

ETL pipelines

Data modelling

Big data

REST APIs

AWS

Scripting

Windows/Linux

Tools

SSIS

AWS DMS

AWS Lambda

ECS

EventBridge

AWS Glue

Beautiful Soup

Selenium

CasperJS

PhantomJS

Node.js

Job description

• Proficient in general data cleaning and transformation (e.g. SQL, pandas, R, etc) to ensure data accuracy and consistency.
• Proficient in building ETL pipeline (eg. SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Event bridge, AWS Glue ,Spring).
• Proficient in database design and various databases (e.g. SQL, PostgreSQL, AWS S3, Athena, mongo db, postgres/gis, mysql, sqlite, volt db, cassandra, etc).
• Experience in cloud technologies such as GPC, GCC (i.e. AWS, Azure, Google Cloud).
• Experience and passion for data engineering in a big data environment using Cloud platforms such as GPC, GCC(i.e. AWS, Azure, Google Cloud).
• Experience with building production-grade data pipelines, ETL/ELT data integration.
• Knowledge about system design, data structure and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.
• Familiar with rest api and web requests/protocols in general.
• Familiar with big data frame works and tools (eg. Hadoop, Spark, Kafka, RabbitMQ).
• Familiar with W3C Document Object Model and customised web scraping (e.g. Beautiful Soup, CasperJS, PhantomJS, Selenium, Nodejs, etc).
• Familiar with data governance policies, access control and security best practices.
• Comfortable in at least one scripting language (eg. SQL, Python).
• Comfortable in both windows and linux development environments.
• Interest in being the bridge between engineering and analytics.