Data Engineer

QUESS ENGINEERING PTE. LTD.; Singapore; Presencial; SGD 90.000 - 140.000; Jornada completa

Descripción de la vacante

QUESS ENGINEERING PTE. LTD. is seeking a data engineering professional to design and implement scalable data solutions across AWS cloud and hybrid environments.

You will build production-grade pipelines using Python, PySpark, SQL and modern data tooling, while contributing to platform modernization and automation initiatives. You will optimise data lakes/lakehouses, support metadata management, and work with AI services, CI/CD pipelines, and observability tools to ensure reliable, secure

Formación

• 3–5+ years of hands-on data pipelines experience.
• Experience with AWS and/or hybrid cloud environments.
• Strong Linux/Unix administration and scripting.
• Understanding ETL/ELT, data integration, data quality.
• Experience with distributed data processing and Lakehouse architectures.
• Proficiency in Git, CI/CD, DevOps, and deployment processes.

Responsabilidades

• Design, develop and implement scalable data and platform solutions across AWS cloud and hybrid/on-premises environments.
• Build, maintain and optimise production-grade data pipelines, platform services and automation frameworks using Python, PySpark, SQL and cloud-native technologies.
• Support and enhance enterprise data platforms, including data lakes, lakehouse architectures, metadata management, search platforms and AI services.
• Develop and maintain CI/CD pipelines, automated testing frameworks and deployment processes to ensure reliable and efficient software delivery.
• Implement, integrate and support technologies such as OpenSearch, Amazon Bedrock, AgentCore, MCP and related platform services.
• Monitor and improve platform reliability, security, performance, observability and cost efficiency.
• Troubleshoot and resolve production issues, perform root-cause analysis and ensure platform stability and operational excellence.
• Collaborate with cross-functional teams to drive platform modernisation, automation and continuous improvement initiatives.

Conocimientos

SQL

Python

PySpark

Apache Spark

Airflow

Git

CI/CD

Linux/Unix

Scripting

Herramientas

AWS

OpenSearch

Kafka/MSK

DBT

Hive/Presto

Iceberg

Datadog

Grafana

Prometheus

Descripción del empleo

Key Responsibilities

• Design, develop and implement scalable data and platform solutions across AWS cloud and hybrid/on-premises environments.

• Build, maintain and optimise production-grade data pipelines, platform services and automation frameworks using Python, PySpark, SQL and cloud-native technologies.

• Support and enhance enterprise data platforms, including data lakes, lakehouse architectures, metadata management, search platforms and AI services.

• Develop and maintain CI/CD pipelines, automated testing frameworks and deployment processes to ensure reliable and efficient software delivery.

• Implement, integrate and support technologies such as OpenSearch, Amazon Bedrock, AgentCore, MCP and related platform services.

• Monitor and improve platform reliability, security, performance, observability and cost efficiency.

• Troubleshoot and resolve production issues, perform root-cause analysis and ensure platform stability and operational excellence.

• Collaborate with cross-functional teams to drive platform modernisation, automation and continuous improvement initiatives.

Mandatory Requirements

• 3–5+ years of hands‑on experience building and supporting production‑grade data pipelines using SQL, Python and cloud technologies.

• Practical experience working with AWS and/or hybrid cloud environments.

• Strong Linux/Unix administration and scripting skills, with experience supporting production systems and troubleshooting operational issues.

• Strong understanding of ETL/ELT processes, data integration, data transformation and data quality management.

• Experience with distributed data processing frameworks and Data Lake/Lakehouse architectures.

• Proficiency in Git, CI/CD pipelines, DevOps practices and production deployment processes.

Technical Requirements

Data Engineering & Processing

• Python, PySpark, SQL, Apache Spark and Airflow.

• ETL/ELT development, data modelling and distributed data processing.

• Data Lake/Lakehouse architectures and table formats such as Apache Iceberg.

AWS & Cloud Platform

• AWS Glue, EMR, S3, Redshift and Lambda.

• Kafka/MSK, DBT, Hive, Trino and Presto.

• OpenSearch, Amazon Bedrock, AgentCore and Model Context Protocol (MCP).

Platform Engineering & Operations

• Linux/Unix administration, shell scripting and production support.

• Git, CI/CD, DevOps practices and Infrastructure as Code (IaC).

• Monitoring, troubleshooting, performance tuning and platform reliability.

Preferred Qualifications

• Experience with enterprise Data Lake and Lakehouse platforms.

• Knowledge of streaming and event‑driven architectures.

• Hands‑on experience with OpenSearch administration, performance optimisation and tuning.

• Familiarity with monitoring and observability tools such as Datadog, Amazon CloudWatch, Grafana or Prometheus.

• Understanding of data governance, security, access controls and Personally Identifiable Information (PII) protection.

• Experience in insurance, banking or financial services environments.