Senior Data Engineer

zhong qi future (singapore) pte. ltd.; Singapore; Híbrido; SGD 120.000 - 180.000; Tempo integral

Resumo da vaga

ZHONG QI FUTURE (SINGAPORE) PTE. LTD. is seeking a Senior Data Engineer to design, build, and operate scalable data pipelines on AWS and Databricks within a cloud data platform.

You will contribute to governance, modelling and data products, collaborating across data engineers, architects and stakeholders. The role requires hands-on Python, PySpark, SQL, and experience with AWS and Databricks, plus leadership in production-ready data solutions in a public-sector context.

Qualificações

• Hands-on Python, PySpark and SQL experience.
• Proficient with AWS and Databricks platforms.
• Experience building scalable data pipelines.
• Strong data governance and security awareness.

Responsabilidades

• Design and implement scalable batch and streaming pipelines on AWS and Databricks.
• Develop lakehouse architectures with Delta Lake and Bronze/Silver/Gold layers.
• Establish data quality controls, lineage and governance.
• Implement CI/CD and IaC for data workflows.
• Monitor performance, reliability and cost, and optimize Spark/SQL workloads.

Conhecimentos

Python

PySpark

SQL

AWS

Databricks

Data pipelines

Data governance

Data modelling

Cloud architectures

Ferramentas

Databricks Unity Catalog

AWS Glue

Lake Formation

Delta Lake

Descrição da vaga

Senior Data Engineer - AWS & Databricks (Cloud Data Platform)

• Contract Duration: 12 Months
• Contract Period: January 2027 - December 2027
• Number of Vacancies: 1
• Working Hours: Monday to Friday, 8:30 AM - 6:00 PM
• Work Arrangement: Hybrid
• Location: Singapore

Job Overview

We are seeking an experienced Senior Data Engineer to join a cloud data platform and architecture team supporting a public sector digital transformation initiative.The successful candidate will be responsible for designing, developing, producing and operating scalable, secure and reliable data pipelines and reusable data products using Amazon Web Services (AWS) and Databricks.You will play a key technical leadership role across the data engineering lifecycle, covering data ingestion, transformation, modelling, orchestration, data quality, governance, monitoring and performance optimisation.Working closely with data architects, analysts, data scientists, platform engineers and business stakeholders, you will translate complex data requirements into robust engineering solutions that support enterprise analytics, reporting and Machine Learning (ML) and Artificial Intelligence (AI) workloads.This position is suited to a technically strong Data Engineer with extensive hands-on experience in Python, PySpark, SQL, AWS and Databricks, coupled with the ability to establish engineering standards, mentor team members and deliver production-ready data solutions.

Key Responsibilities

Data Engineering & Pipeline Architecture

• Lead the design, development and implementation of scalable batch and streaming data pipelines across AWS and Databricks environments.
• Build and maintain robust data ingestion, validation, transformation and publishing processes for structured, semi-structured and unstructured data.
• Integrate data from multiple sources, including databases, files, APIs and event streams.
• Implement reliable full-load, incremental loading and Change Data Capture (CDC) mechanisms, including data replay, recovery and backfill processes.
• Ensure data pipelines are maintainable, reusable, scalable and suitable for production operations.

Lakehouse Architecture & Data Product Development

• Design and implement modern Lakehouse architectures using Delta Lake and layered data processing patterns, including Bronze, Silver and Gold data layers.
• Develop reusable data models, curated datasets and domain-oriented data products to support enterprise analytics and downstream applications.
• Establish clear data ownership, definitions, quality expectations and service requirements for published data products.
• Apply appropriate dimensional and domain data modelling techniques to improve data accessibility, consistency and usability.
• Promote reusable engineering components and standardized data solutions rather than developing isolated, project-specific pipelines.

Data Quality, Governance & Management

• Establish automated data quality controls, schema enforcement, validation and reconciliation processes.
• Implement data quality checks covering completeness, accuracy, consistency, uniqueness, timeliness and referential integrity.
• Develop and maintain technical metadata, business definitions, data classification and end-to-end data lineage.
• Implement data governance and access management practices using platforms and tools such as Databricks Unity Catalog, AWS Glue Data Catalog and Lake Formation.
• Ensure data assets are appropriately documented, governed and accessible in accordance with established policies and security requirements.

Workflow Orchestration & CI/CD Automation

• Design and implement automated data workflows using Databricks Workflows, Lakeflow and relevant AWS services.
• Establish automated testing, deployment and environment promotion processes across development, testing and production environments.
• Integrate Git-based development practices, Continuous Integration and Continuous Deployment (CI/CD), and Infrastructure as Code (IaC).
• Develop parameterized and reusable workflows to improve deployment consistency, operational efficiency and maintainability.
• Ensure appropriate pipeline scheduling, dependency management, retry mechanisms and recovery procedures.

Monitoring, Reliability & Performance Optimization

• Establish monitoring, logging, alerting and operational dashboards to track data pipeline performance and reliability.
• Monitor pipeline health, data freshness, processing latency, throughput and data quality issues.
• Perform troubleshooting, incident investigation and root-cause analysis, implementing appropriate corrective and preventive measures.
• Optimize Apache Spark and SQL workloads, including partitioning, clustering, caching, compaction, file sizing and cluster configurations.
• Improve processing efficiency, workload scheduling and overall AWS and Databricks infrastructure cost effectiveness.
• Develop and maintain operational runbooks and support documentation to ensure prod