Data Engineer

WPH DIGITAL PTE. LTD.; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A technology company in Singapore is looking for a Data Engineer to design and develop data systems. You will build ETL pipelines, perform data extraction, and work with cloud technologies like AWS and Azure. Ideal candidates should have a relevant degree and experience in database design and big data frameworks. This position requires collaboration with various teams and a passion for bridging engineering and analytics.

Qualifications

• Degree or Diploma in Computer Science, Computer or Electronics Engineering, Information Technology or related disciplines.
• Proficient in building ETL pipelines with various tools.
• Experience with cloud technologies like AWS, Azure, and Google Cloud.

Responsibilities

• Design, develop and deploy data tables, views and marts.
• Perform data extraction, cleaning, transformation, and flow.
• Integrate and collate data silos in a scalable manner.

Skills

ETL pipeline building

Database design

Cloud technologies

Big data frameworks

Scripting languages

Education

Degree or Diploma in Computer Science or related field

Tools

SQL Server Integration Services (SSIS)

AWS Database Migration Services (DMS)

Hadoop

Spark

BeautifulSoup

Job description

Responsibility

• Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation, and flow. Web scraping may be also a part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner which is both scalable and compliant.
• Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analyst to build scalable datadriven products.
• Be responsible for developing backend APIs & working on databases to support the applications.
• Work in an Agile Environment that practices Continuous Integration and Delivery.
• Work closely with fellow developers through pair programming and code review process.

Experiences & Skills Needed

• Degree or Diploma in Computer Science, Computer or Electronics Engineering, Information Technology or related disciplines.
• Proficient in building ETL pipeline (eg. SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Eventbridge, AWS Glue, Spring).
• Proficient in database design and various databases (e.g. SQL, PostgreSQL, AWS S3, Athena, mongodb, postgres/gis, mysql, sqlite, voltdb, cassandra, etc).
• Experience in cloud technologies such as GPC, GCC (i.e. AWS, Azure, Google Cloud).
• Experience and passion for data engineering in a big data environment using
• Cloud platforms such as GPC, GCC (i.e. AWS, Azure, Google Cloud).
• Experience with building production-grade data pipelines, ETL/ELT data integration.
• Knowledge about system design, data structure and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.
• Familiar with rest api and web requests/protocols in general.
• Familiar with big data frameworks and tools (eg. Hadoop, Spark, Kafka,RabbitMQ).
• Familiar with W3C Document Object Model and customized web scraping (e.g. BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs, etc).
• Familiar with data governance policies, access control and security best practices.
• Comfortable in at least one scripting language (eg. SQL,Python).
• Comfortable in both windows and linux development environments.
• Interest in being the bridge between engineering and analytics