Senior Data Engineer - #1732

JOBSTER PRIVATE LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

JOBSTER PRIVATE LTD. seeks a seasoned Data Engineer to design and deliver maintainable batch and streaming data pipelines on AWS and Databricks.

You will build bronze-to-gold data models, enforce data quality and governance, and optimise performance and cost across large-scale datasets. You will lead data integration from databases, files and APIs, implement CD/CI and IaC, and craft reliable monitoring, dashboards and runbooks to ensure data freshness and downstream reliability for business

Qualifications

• Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline.

Responsibilities

• Lead the design and implementation of maintainable batch and streaming pipelines in AWS and Databricks.
• Design Bronze, Silver and Gold data layers using Delta Lake patterns.
• Implement reliable ingestion from databases, files, APIs and event streams.
• Establish schema enforcement, data contracts and automated validation.

Skills

Python

PySpark

SQL

Delta Lake

Databricks

Spark

AWS

CI/CD

Terraform

Observability

Security basics

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Apache Spark

Delta Lake

Terraform

Git

Job description

Key Responsibilities

• Data Pipeline Architecture and Delivery: Lead the design and implementation of maintainable batch and streaming pipelines that ingest, validate, transform and publish structured, semi-structured and unstructured data across AWS and Databricks.
• Lakehouse and Data Product Engineering: Design Bronze, Silver and Gold data layers using Delta Lake or equivalent patterns; develop reusable domain data models, curated datasets and data products with explicit ownership, definitions and service expectations.
• Source Integration and Incremental Processing: Implement reliable ingestion from databases, files, APIs and event streams using full-load, incremental, change-data-capture and replayable processing patterns appropriate to each source.
• Data Quality and Contracts: Establish schema enforcement, data contracts, automated validation, reconciliation and anomaly checks covering completeness, validity, consistency, uniqueness, timeliness and referential integrity.
• Orchestration and Deployment Automation: Build parameterised workflows using Databricks Workflows or Lakeflow and relevant AWS services; integrate automated testing, nfrastructure as code and CI/CD controls across development, test and production environments.
• Metadata, Lineage and Governance: Implement technical metadata, business definitions, end-to-end lineage, classification, ownership and access controls using Unity Catalog, AWS Glue Data Catalog, Lake Formation or approved platform capabilities.
• Pipeline Monitoring and Reliability: Define dashboards, alerts, logging, operational metrics and runbooks for pipeline health, freshness, data-quality failures, latency, throughput and downstream impact; lead troubleshooting and root-cause analysis.
• Performance and Cost Optimisation: Optimise Spark and SQL workloads, file sizes, partitioning, clustering, caching, compaction, cluster configuration and workload scheduling to improve runtime, reliability and AWS/Databricks cost efficiency.
• Security and Data Protection: Apply least-privilege IAM, encryption, secrets management, environment separation, audit logging, sensitive-data controls and secure data-sharing patterns in line with applicable agency requirements.
• Technical Leadership and Quality: Set engineering standards, review designs and code, mentor engineers, manage technical debt and promote reusable components rather than project-specific pipelines.
• Stakeholder Collaboration: Work with data owners, stewards, analysts, data scientists, architects, product teams and governance stakeholders to translate analytical and operational needs into implementable data specifications.
• Documentation and Knowledge Transfer: Maintain architecture decisions, source-to-target mappings, data models, contracts, operating procedures and support documentation, and transfer knowledge to the internal team throughout the engagement.

Requirements & Qualifications

A. Educational Qualification & Experience

• Education: Degree in Computer Science, Software Engineering, Data Engineering, Information Systems or a related technical discipline.
• Experience : Typically 7-10 years of relevant data or software engineering experience, including hands-on design and operation of production data pipelines and at least 2 years working with cloud-based data platforms. Candidates should demonstrate technical leadership through architecture, engineering standards, code review, mentoring or ownership of critical data services.

B. Technical & Professional Competencies

Essential (Must-Have)

• Strong Python, PySpark and SQL capability, including distributed processing, complex transformations, query optimisation, debugging and automated testing.
• Hands-on experience with Databricks, Apache Spark and Delta Lake for production-scale batch or streaming data engineering.
• Hands-on AWS data engineering experience with a relevant mix of S3, Glue, Lake Formation, IAM, KMS, Lambda, Step Functions, EventBridge, CloudWatch, Redshift and MSK/Kinesis.
• Experience designing lakehouse or data-lake architectures, including layered data patterns, dimensional or domain data modelling and reusable curated datasets.
• Experience implementing incremental ingestion, change data capture, idempotency, late-arriving data handling, backfills, replay and recovery.
• Experience implementing data quality, schema validation, reconciliation, metadata and lineage as automated parts of production pipelines.
• Experience with workflow orchestration, Git-based development, CI/CD, infrastructure as code such as Terraform, and environment promotion controls.
• Ability to implement pipeline observability, operational dashboards, alerting, runbooks, incident diagnosis and service recovery.
• Working knowledge of cloud security, least-privilege access, network boundaries, encryption, secret management, data classification and audit logging.
• Ability to review technical designs and code, establish engineering standards, and guide le