Data Engineer

ELLIOTT MOSS CONSULTING PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

ELLIOTT MOSS CONSULTING PTE. LTD.

in Singapore seeks an experienced Data Engineer to lead the Gold layer Data Mart design on the Unified Data Platform (Databricks), building scalable pipelines for Power BI reporting and acting as bridge between Singapore design team and offshore engineers. You will own ETL/ELT development, leverage Delta Lake, Delta Live Tables, Unity Catalog, and maintain data quality, governance, and performance.

Qualifications

• 5+ years of data engineering with ETL/ELT focus.
• Experience with Databricks platform and Delta Lake.
• Proficiency in PySpark and Databricks SQL for large-scale transformations.
• Strong understanding of star schema, SCD Type 2, and dimensional modelling.
• Experience implementing data quality frameworks and SIT/UAT support.

Responsibilities

• Design Gold layer Data Mart tables and views in Databricks per STTM.
• Implement Silver-to-Gold transformations using DLT, Databricks Workflows, and PySpark.
• Create Delta table DDL in Unity Catalog with schema and comments.
• Apply SCD Type 2 and aggregation logic per grain specs.
• Configure partitioning, Z-Order, and clustering for performance.
• Develop data quality checks and Delta Live Tables DQ rules.
• Schedule and manage Databricks Workflow DAGs for pipelines.

Skills

Databricks

PySpark

Delta Lake

Delta Live Tables

Databricks SQL

Unity Catalog

ETL/ELT

Dimensional modelling

SCD Type 2

Data quality frameworks

Git

CI/CD for Databricks

Power BI integration

Tools

Databricks Workflows

Delta Live Tables

Databricks Unity Catalog access control

Delta Lake

Job description

Job Description

• We are looking for an experienced Data Engineer to join the delivery team for the EDW Reporting Modernisation project.
• You will be responsible for designing and building the Gold layer Data Mart tables, transformation pipelines, and associated views on the Unified Data Platform (Databricks), working closely with the Tech Lead and Data Modeler to deliver a robust and scalable reporting data layer that caters to Power BI reports.
• As the onshore Data Engineer, you will serve as the technical bridge between the Singapore-based design team and the offshore engineering squad, providing day-to-day technical direction, code review, and quality oversight across the full Data Mart build.

Key Responsibilities

Design and Development

• Develop Gold layer Data Mart tables and associated database views in Databricks in accordance with signed-off Source-to-Target Mapping (STTM) documents and design specifications
• Implement Silver-to-Gold transformation pipelines using Databricks Workflows, Delta Live Tables (DLT), and PySpark
• Create Delta table DDL in Unity Catalog, including schema definition, table properties, and column-level comments
• Implement SCD Type 2 merge logic for dimension tables and aggregation logic for fact tables per the grain specification
• Configure table partitioning, Z-ORDER clustering, and Liquid Clustering strategies for query performance optimisation
• Develop and implement data quality expectations using Delta Live Tables DQ framework per the DQ thresholds defined in the design specification
• Schedule and configure Databricks Workflow job DAGs for all pipeline runs, including watermark-based incremental load logic

Testing and Quality Assurance

• Conduct unit testing for each Data Mart table upon build completion — covering row count validation, null checks, duplicate grain checks, and transformation logic verification
• Execute system integration test (SIT) scenarios for all 60 Data Mart tables, including aggregated value checks, referential integrity validation, and SCD integrity checks
• Log, investigate, and resolve defects identified during SIT and UAT, performing root cause analysis and documenting fixes
• Review and validate unit test outputs from Pune Data Engineers before test results are submitted for Tech Lead review

Offshore Squad Technical Leadership

• Provide day-to-day technical direction and coding guidance to the 2 offshore-based Data Engineers
• Conduct code reviews for all pipeline and DDL code produced by the offshore squad before submission for Tech Lead sign-off
• Ensure offshore squad adherence to agreed coding standards, naming conventions, Unity Catalog governance rules, and development best practices
• Facilitate knowledge sharing and technical problem-solving with the Pune squad on complex transformation and merge logic

Deployment and Handover

• Execute Data Mart pipeline and table deployment to the production Databricks environment per the approved deployment runbook
• Validate the first production pipeline run and confirm data freshness post-deployment
• Contribute to the operational runbook covering pipeline architecture, job schedules, alert thresholds, and common failure scenarios
• Support knowledge transfer sessions with the Singtel IT operations team during the handover phase

Required Skills and Experience

• 5+ years of experience in data engineering with a strong focus on ETL/ELT pipeline development and dimensional data modelling
• Hands-on experience with Databricks — including Delta Lake, Delta Live Tables, Databricks Workflows, Unity Catalog, and Databricks SQL
• Proficiency in PySpark and Databricks SQL for large-scale data transformation
• Strong understanding of dimensional modelling concepts — star schema, SCD Type 2, surrogate key design, fact and dimension table design
• Experience implementing data quality frameworks and reconciliation testing
• Familiarity with Source-to-Target Mapping (STTM) and translating design specifications into production-grade pipelines
• Experience with incremental load patterns — watermark-based, partition-based, or CDC-driven
• Proficiency in Git-based version control for collaborative development
• Experience with Databricks Unity Catalog access control, table tagging, and lineage tracking
• Exposure to Oracle-to-Databricks migration projects or similar platform modernisation programmes
• Familiarity with Power BI Semantic Models and how Gold layer table design impacts downstream DAX measure performance
• Experience with Databricks Assistant or AI-assisted code generation tools for accelerated pipeline development
• Knowledge of CI/CD pipeline setup for Databricks notebook or YAML