Senior Data Engineer: Python, Spark, Databricks & AWS

Tap Growth ai; Singapore; On-site; SGD 90,000 - 130,000; Full time

Job summary

Tap Growth ai in Singapore is seeking a Data Engineer/Senior Data Engineer with 5+ years of experience to design, build, and maintain robust data pipelines and architectures. You will ensure reliability, scalability, and performance of data infrastructure to support BI and analytics initiatives.

Work from office in Singapore. The role focuses on lakehouse architectures, streaming/batch processing, and end-to-end data engineering workflows using Python, PySpark, Spark, Delta Lake, and AWS

Qualifications

• Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query optimisation, debugging and automated testing.
• Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering.
• Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch, Redshift and MSK/Kinesis.
• Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional and domain data modelling and reusable curated datasets.
• Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling, backfills, replay and recovery.
• Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated parts of production pipelines.
• Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform, and environment promotion controls.
• Ability to implement pipeline observability, operational dashboards, alerting, runbooks, incident diagnosis and service recovery.
• Working knowledge of cloud security, least-privilege access, network boundaries, encryption, secret management, data classification and audit logging.
• Ability to review technical designs and code, establish engineering standards, and guide less-experienced engineers.

Skills

Python

PySpark

SQL

Databricks

Apache Spark

Delta Lake

AWS

S3

Glue

Lake Formation

IAM

KMS

Lambda

Step Functions

EventBridge

CloudWatch

Redshift

Kinesis

Tools

Terraform

Job description

Tap Growth ai in Singapore is seeking a Data Engineer/Senior Data Engineer with 5+ years of experience to design, build, and maintain robust data pipelines and architectures. You will ensure reliability, scalability, and performance of data infrastructure to support BI and analytics initiatives.

Work from office in Singapore. The role focuses on lakehouse architectures, streaming/batch processing, and end-to-end data engineering workflows using Python, PySpark, Spark, Delta Lake, and AWS