Senior Data Engineer

BASIL TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

BASIL TECHNOLOGIES PTE. LTD. in Singapore seeks a data engineer to design, build and maintain data tables, marts, and pipelines across data warehouses, data lake and data virtualization. You will collaborate with cross-functional teams to deliver scalable data-driven products and APIs, in an Agile environment with CI/CD.

The role emphasizes data extraction, web-scraping, and cloud-based architectures on AWS, Azure, and Google Cloud, with a focus on scalable pipelines and data governance.

Qualifications

• Proficient in data cleaning and transformation using SQL, pandas, and R.

Responsibilities

• Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation, and flow. Web scraping may also be part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large-scale batch and real-time data pipelines with data processing frameworks.
• Integrate and collate data silos in a scalable and compliant manner.
• Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analyst to build scalable data-driven products.
• Responsible for developing backend APIs and working on databases to support the applications.
• Work in an Agile Environment that practices Continuous Integration and Delivery.
• Work closely with fellow developers through pair programming and code review process.
• The team is expected to perform Data Warehousing tasks, mainly in AWS GCC, and manage APIs.

Skills

SQL

pandas

Python

REST API

data pipelines

data engineering

Tools

SSIS

AWS DMS

Python

AWS Lambda

ECS

EventBridge

AWS Glue

Spring

PostgreSQL

MongoDB

S3

Athena

MySQL

SQLite

VoltDB

Cassandra

Hadoop

Spark

Kafka

RabbitMQ

Job description

Job Description

• Design, develop and deploy data tables, views and marts in data warehouses, operational data store, data lake and data virtualization.
• Perform data extraction, cleaning, transformation, and flow. Web scraping may also be part of the work scope in data extraction.
• Design, build, launch and maintain efficient and reliable large‑scale batch and real‑time data pipelines with data processing frameworks.
• Integrate and collate data silos in a manner that is both scalable and compliant.
• Collaborate with Project Manager, Data Architect, Business Analysts, Frontend Developers, Designers and Data Analyst to build scalable data‑driven products.
• Responsible for developing backend APIs and working on databases to support the applications.
• Work in an Agile Environment that practices Continuous Integration and Delivery.
• Work closely with fellow developers through pair programming and code review process.
• The team is expected to perform Data Warehousing tasks, mainly in AWS GCC, and manage APIs.

Qualifications

• Proficient in general data cleaning and transformation (e.g., SQL, pandas, R) to ensure data accuracy and consistency.
• Proficient in building ETL pipelines (e.g., SQL Server Integration Services (SSIS), AWS Database Migration Services (DMS), Python, AWS Lambda, ECS container tasks, EventBridge, AWS Glue, Spring).
• Proficient in database design and various databases (e.g., SQL, PostgreSQL, AWS S3, Athena, MongoDB, PostgreSQL/GIS, MySQL, SQLite, VoltDB, Cassandra).
• Experience in cloud technologies such as AWS, Azure, Google Cloud.
• Experience and passion for data engineering in a big‑data environment using cloud platforms such as AWS, Azure, Google Cloud.
• Experience with building production‑grade data pipelines, ETL/ELT data integration.
• Knowledge about system design, data structure and algorithms.
• Familiar with data modelling, data access, and data storage infrastructure like Data Mart, Data Lake, Data Virtualisation and Data Warehouse for efficient storage and retrieval.
• Familiar with REST API and web requests/protocols in general.
• Familiar with big‑data frameworks and tools (e.g., Hadoop, Spark, Kafka, RabbitMQ).
• Familiar with W3C Document Object Model and custom web scraping (e.g., BeautifulSoup, CasperJS, PhantomJS, Selenium, Node.js).
• Familiar with data governance policies, access control and security best practices.
• Comfortable in at least one scripting language (e.g., SQL, Python).
• Comfortable in both Windows and Linux development environments.
• Interest in being the bridge between engineering and analytics.