T&T Consultant – Engineering, Data Engineer

Jobtailor; Singapore; On-site; SGD 90,000 - 150,000; Full time

Job summary

Jobtailor in Singapore is seeking a Data Engineer to design, build and deploy data tables, views and data marts across data warehouses, data lake, data virtualization and ODS.

You will collaborate with cross-functional teams to ensure scalable, reliable data pipelines, perform data extraction, cleaning and transformation, and contribute to production-grade backend APIs and data storage strategies.

Qualifications

• 4-6 years of data engineering experience.
• Strong data cleaning and transformation skills (SQL, pandas, R).
• Experience building ETL pipelines using SSIS, AWS DMS, Python, AWS Lambda, ECS, Eventbridge, AWS Glue.

Responsibilities

• Design, develop and deploy data tables, views and marts in data warehouses, ODS, data lake and data virtualization.
• Build and maintain large-scale batch and real-time data pipelines.

Skills

SQL

Python

Data Cleaning and Transformation

ETL/ELT Data Integration

Data Modeling

Big Data Frameworks

Web Scraping

Database Management

Tools

AWS Lambda

AWS Glue

SSIS

Apache Kafka

RabbitMQ

Eventbridge

ECS Container

W3C DOM

Job description

• Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation, and flow.
• Web scraping may be also a part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner which is both scalable and compliant.
• Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analyst to build scalable data-driven products.
• Be responsible for developing backend APIs & working on databases to support the applications.
• Work in an Agile Environment that practices Continuous Integration and Delivery.
• Work closely with fellow developers through pair programming and code review process.

Requirements

• Has 4-6 years of relevant working experience as a data engineer.
• Proficient in general data cleaning and transformation (e.g. SQL, pandas, R, etc) to ensure data accuracy and consistency.
• Proficient in building ETL pipeline (eg. SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS Container task, Eventbridge, AWS Glue, Spring).
• Proficient in database design and various databases (e.g. SQL, PostgreSQL, AWS S3, Athena, mongodb, postgres/gis, mysql, sqlite, voltdb, cassandra, etc).
• Experience in cloud technologies such as GPC, GCC (i.e. AWS, Azure, Google Cloud).
• Experience and passion for data engineering in a big data environment using Cloud platforms such as GPC, GCC (i.e. AWS, Azure, Google Cloud).
• Experience with building production-grade data pipelines, ETL/ELT data integration.
• Knowledge about system design, data structure and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.
• Familiar with rest api and web requests/protocols in general.
• Familiar with big data frameworks and tools (eg. Hadoop, Spark, Kafka,RabbitMQ).
• Familiar with W3C Document Object Model and customized web scraping (e.g. BeautifulSoup, CasperJS, PhantomJS, Selenium, Nodejs, etc).
• Familiar with data governance policies, access control and security best practices.
• Comfortable in at least one scripting language (eg. SQL,Python).
• Comfortable in both windows and linux development environments.

Core Competencies

Demonstrates expertise in designing and deploying data pipelines, data warehousing, and ETL processes using cloud technologies. Proficient in data cleaning, transformation, and integration while ensuring compliance and scalability.

Hard Skills

• SQL
• Python
• Data Cleaning and Transformation
• ETL/ELT Data Integration
• Data Modeling
• Big Data Frameworks (Hadoop, Spark)
• Web Scraping (BeautifulSoup, Selenium)
• Database Management (PostgreSQL, MySQL)
• Data Warehouse Design
• Data Lake Implementation

Soft Skills

• Collaboration
• Problem-Solving
• Agile Methodologies
• Communication

Industry Keywords

• Data Engineering
• Data Governance Policies
• Data Access Control
• Big Data Environment
• Data Virtualization

Tools & Technologies

• AWS Lambda
• AWS Glue
• SQL Server Integration Services (SSIS)
• Apache Kafka
• RabbitMQ
• Eventbridge
• ECS Container
• W3C Document Object Model