Data Engineer (ETL, Snowflake, AWS Glue)

QUESS SELECTION & SERVICES PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

QUESS SELECTION & SERVICES PTE. LTD. in Singapore is seeking a Lead Data Engineer to drive data engineering efforts, oversee BAU operations, and ensure scalable data pipelines across the data lake and data warehouse.

You will partner with stakeholders to design data structures and reliable ingestion processes, guiding migration to cloud-native tools like Snowflake, Spark, and Glue. The role requires hands-on SQL/Python/PySpark expertise, strong AWS knowledge, and experience leading teams to

Qualifications

• Experience with AWS Glue, Airflow, Lambda, S3 and Snowflake.
• Proficient in SQL and Python / PySpark.
• Experience with CI/CD pipelines; production support.

Responsibilities

• Lead a team of data engineers in managing day-to-day BAU operations, production support, incident management, and platform stability.
• Develop tools to improve data flows between internal/external systems and the data lake/warehouse.
• Build robust and reproducible data ingest pipelines to collect, clean, harmonize, merge, and consolidate data sources.
• Collaborate with Data Scientists, Architect and Team on several Data Analytics projects.
• Lead the migration of data transformation jobs to Snowflake, Spark, Python on Glue, etc.
• Document processes and maintain datasets; improve data quality and efficiency.

Skills

Team leadership

AWS

Snowflake

Airflow

MWAA

SQL

Python

PySpark

CI/CD

Cloud optimization

Data strategy

Tools

AWS Glue

Lambda

S3

Snowflake

Job description

Roles & Responsibilities:

• Develop tools to improve data flows between internal/external systems and the data lake/warehouse.
• Work with stakeholders to understand needs for data structure, availability, scalability, and accessibility.
• Build robust and reproducible data ingest pipelines to collect, clean, harmonize, merge, and consolidate data sources.
• Understanding existing data applications and infrastructure architecture
• Build and support new data feeds for various Data Management layers and Data Lakes
• Evaluate business needs and requirements.
• Support migration of existing data transformation jobs in Oracle, and MS-SQL to Snowflake.
• Lead the migration of the existing data transformation jobs in Oracle, Hive, Impala etc. into Spark, Python on Glue etc.
• Able to document the processes and steps.
• Develop and maintain datasets.
• Improve data quality and efficiency.
• Lead Business requirements and deliver accordingly.
• Collaborate with Data Scientists, Architect and Team on several Data Analytics projects.
• Collaborate with DevOps Engineer to improve system deployment and monitoring process.
• Experience in critical production support and how the BAU functions is preferred.

Key Requirements:

• Lead a team of data engineers in managing day-to-day BAU operations, production support, incident management, and platform stability.
• Strong AWS knowledge in terms of designing new architecture and providing optimized solutions for existing ones. (S3, Glue, DMS, MWAA, AMS, IAM).
• In-depth knowledge with respect to Snowflake and its architecture.
• Prefer prior Experience in BAU environment.
• Good knowledge on Airflow and MWAA.
• Hands-on experience in SQL/Python/Pyspark.
• Expertise in optimizing techniques in cloud environments.
• Should have the vision on data strategy and able to deliver the same.

Must Have: AWS - Glue, Airflow, Lambda, S3 , Snowflake (Related Cloud DB) SQL ,Python/ Spark,CI/CD Good To have : ECS,GIT, Oracle