Senior Data Engineer – AWS / Databricks / PySpark

ASTEK SINGAPORE INNOVATION TECHNOLOGY PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Astek Singapore Innovation Technology Pte. Ltd. is seeking a Senior Data Engineer to design and maintain scalable data pipelines on AWS and Databricks.

You will lead data lakehouse architecture, ensure data quality and governance, and collaborate with data scientists and analysts to deliver reliable data products. The role requires strong Python/PySpark/SQL skills, Celestial cloud service experience, and leadership to mentor junior engineers.

Qualifications

• 6–10 years of data/software engineering experience, incl. 2+ years on cloud data platforms.
• Hands-on with Python, PySpark, SQL, Spark, Databricks and Delta Lake.
• Experience with AWS data services: S3, Glue, Lake Formation, IAM, KMS, Redshift, MSK/Kinesis.
• Design of data lake/lakehouse architectures and data models.
• Strong ETL/ELT, CDC, data quality, metadata and lineage skills.
• Automation, CI/CD, testing and infrastructure as code (Terraform).
• Data pipeline monitoring, observability and performance tuning.
• Cloud security, encryption, access management and governance.
• Proven ability to mentor engineers and lead code reviews.

Responsibilities

• Design, develop and optimise scalable batch and streaming pipelines on AWS, Databricks, Python, PySpark and SQL.
• Build lakehouse architectures with Bronze-Silver-Gold layers and reusable data products.
• Ingest data from databases, APIs, files and event streams; support CDC and backfills.
• Establish automated data quality checks, schema validation, lineage and metadata management.
• Develop workflows with Databricks Workflows, Lakeflow and AWS services; implement CI/CD and IaC.
• Enforce data governance, access controls and security using Unity Catalog and Lake Formation.
• Monitor performance, troubleshoot production issues and optimise Spark/SQL workloads.
• Provide technical leadership, code reviews and mentoring to engineers.
• Collaborate with data scientists, analysts and stakeholders to deliver reliable data solutions.
• Maintain technical documentation and knowledge transfer materials.

Skills

Python

PySpark

SQL

Apache Spark

Databricks

Delta Lake

ETL/ELT

CDC

Data quality

Data lineage

CI/CD

Terraform

Git

Cloud security

Education

Bachelor's degree in Computer Science or related field

Tools

AWS

S3

Glue

Lake Formation

IAM

KMS

Lambda

Step Functions

EventBridge

CloudWatch

Redshift

MSK/Kinesis

Databricks

Delta Lake

Delta Live Tables

Auto Loader

Structured Streaming

Databricks Unity Catalog

Lakeflow

Airflow

dbt

Job description

Senior Data Engineer – AWS / Databricks / PySpark

Role Overview

We are seeking an experienced Senior Data Engineer to design, develop and maintain scalable data pipelines and reusable data products on an AWS and Databricks-based data platform. The role involves technical leadership in data engineering, lakehouse architecture, data quality, governance and performance optimisation to support analytics, reporting and AI/ML workloads.

Key Responsibilities

• Design, develop and optimise scalable batch and streaming data pipelines using AWS, Databricks, Python, PySpark and SQL.

• Build and maintain lakehouse architectures using Delta Lake, including Bronze, Silver and Gold data layers, curated datasets and reusable data products.

• Implement data ingestion from databases, APIs, files and event streams, including incremental loading, Change Data Capture (CDC), backfills and recovery.

• Establish automated data quality checks, schema validation, reconciliation, metadata management and data lineage.

• Develop and automate data workflows using Databricks Workflows, Lakeflow and AWS services, incorporating CI/CD, automated testing and infrastructure as code.

• Implement data governance, access controls and security measures using Unity Catalog, AWS Glue Data Catalog, Lake Formation and IAM.

• Monitor pipeline performance, troubleshoot production issues and optimise Spark/SQL workloads for scalability, reliability and cost efficiency.

• Lead technical design and code reviews, establish engineering standards and mentor junior engineers.

• Collaborate with data scientists, analysts, architects and business stakeholders to deliver reliable data solutions.

• Maintain technical documentation, data models, operational procedures and knowledge transfer materials.

Requirements

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related discipline.

• 6–10 years of data/software engineering experience, including at least 2 years working with cloud-based data platforms.

• Strong hands-on experience with Python, PySpark, SQL, Apache Spark, Databricks and Delta Lake.

• Proficiency in AWS data services such as S3, Glue, Lake Formation, Lambda, Step Functions, EventBridge, CloudWatch, Redshift, MSK/Kinesis, IAM and KMS.

• Experience designing data lake/lakehouse architectures, data models and production-grade batch or streaming pipelines.

• Strong understanding of ETL/ELT, CDC, incremental ingestion, data quality, schema validation, metadata and lineage.

• Experience with workflow orchestration, Git, CI/CD, automated testing and infrastructure as code, preferably Terraform.

• Knowledge of data pipeline monitoring, observability, troubleshooting and performance optimisation.

• Familiarity with cloud security, encryption, access management and data governance.

• Proven ability to provide technical guidance, conduct code reviews and mentor engineers.

Good to Have

• Experience with Databricks Unity Catalog, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, Delta Sharing and Databricks Asset Bundles.

• Familiarity with Apache Airflow, dbt, Kafka, AWS MSK/Kinesis and AWS DMS.

• Knowledge of data contracts, data products, semantic layers, master/reference data and data mesh concepts.

• Experience supporting ML/AI data pipelines, feature engineering, feature stores or MLOps.

• Exposure to Singapore public-sector environments, including IM8, VAPT, data classification and cross-agency data governance.