Data Engineer

Riskdata Consulting; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Riskdata Consulting is seeking a data engineer to design and implement data tables, views, and scalable pipelines across data warehouses, lakes and data virtualisation. You will collaborate across product, analytics and frontend teams to deliver data-driven products.

The role emphasizes building production-grade ETL/ELT workflows, integrating diverse data sources, and delivering reliable APIs for applications in an Agile CI/CD setting.

Qualifications

• Experience with data cleaning and transformation using SQL, Pandas or R.
• Experience building ETL pipelines with SSIS, AWS DMS, Python, AWS Lambda, ECS, AWS Glue.
• Experience with SQL, PostgreSQL and diverse databases.
• Cloud experience with AWS, Azure or Google Cloud.
• Proven record of production-grade data pipelines and data integration.
• Knowledge of data modelling and data governance.
• Familiar with web scraping and REST APIs.
• Familiar with big data tools like Hadoop, Spark and Kafka.
• Comfortable on Windows and Linux, and with scripting (SQL, Python).
• Interest bridging engineering and analytics.

Responsibilities

• Design, develop and deploy data tables, views and marts in data warehouses and data lake.
• Build scalable batch and real-time data pipelines.
• Integrate data silos in scalable and compliant ways.
• Collaborate with PM, Data Architect, Analysts, Frontend Devs and Designers.
• Develop backend APIs and work on databases to support applications.
• Work in Agile CI/CD environment.
• Pair-programming and code reviews with teammates.

Skills

SQL

Pandas

R

Data cleaning

Data transformation

Data modelling

Big data

REST API

Scripting

Algorithms

Tools

SSIS

AWS DMS

Python

AWS Lambda

ECS

AWS Glue

Spring

BeautifulSoup

Selenium

CasperJS

PhantomJS

Node.js

Job description

Description

• Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation, and flow. Web scraping may be also a part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner which is both scalable and compliant.
• Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analyst to build scalable data driven products.
• Be responsible for developing backend APIs & working on databases to support the applications.
• Work in an Agile Environment that practices Continuous Integration and Delivery.
• Work closely with fellow developers through pair programming and code review process.

Requirements

• Proficient in general data cleaning and transformation (e.g. SQL, pandas, R, etc) to ensure data accuracy and consistency.
• Proficient in building ETL pipeline (eg. SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Eventbridge, AWS Glue, Spring).
• Proficient in database design and various databases (e.g. SQL, PostgreSQL, AWS S3, Athena, mongodb, postgres/gis, mysql, sqlite, voltdb, cassandra, etc).
• Experience in cloud technologies such as GPC, GCC (i.e. AWS, Azure, Google Cloud).
• Experience and passion for data engineering in a big data environment using Cloud platforms such as GPC, GCC (i.e. AWS, Azure, Google Cloud).
• Experience with building production-grade data pipelines, ETL/ELT data integration.
• Knowledge about system design, data structure and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.
• Familiar with rest api and web requests/protocols in general.
• Familiar with big data frameworks and tools (eg. Hadoop, Spark, Kafka,RabbitMQ).
• Familiar with W3C Document Object Model and customized web scraping (e.g. BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs, etc).
• Familiar with data governance policies, access control and security best practices.
• Comfortable in at least one scripting language (eg. SQL,Python).
• Comfortable in both windows and linux development environments.
• Interest in being the bridge between engineering and analytics.