Senior Data Engineer

OPENSOURCE TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 70,000 - 90,000; Full time

Job summary

A leading technology firm in Singapore seeks a Senior Data Engineer to design and maintain high-performance data pipelines for financial workloads. This role requires expertise in Python, PySpark, and cloud platforms like Azure. The ideal candidate has 6-10 years of experience in data engineering, particularly in banking or financial services. Responsibilities include data modernization and collaboration with data scientists. Competitive package offered.

Qualifications

• 6–10 years of experience in data engineering, with at least 3 years in BFSI.
• Proven experience building real-time and batch data pipelines on Azure or AWS.
• Exposure to regulatory data models (MAS 610, Basel III, IFRS 9/17).

Responsibilities

• Design, implement, and optimize ETL/ELT data pipelines using Spark or Databricks.
• Build and operationalize real-time streaming pipelines using Kafka.
• Integrate and transform data across various systems.

Skills

Python

PySpark

SQL

Scala

Apache Airflow

Azure Data Factory

Kafka

Databricks

Snowflake

Education

Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field

Tools

Terraform

Azure DevOps

GitHub Actions

Job description

Role Overview

The Senior Data Engineer is responsible for designing, building, and maintaining large-scale, secure, and high-performance data pipelines supporting critical Financial Services workloads.

The role focuses on data modernization, regulatory data aggregation, and AI/ML enablement across domains such as Core Banking, Payments, Risk, Treasury, and Regulatory Reporting.

Key Responsibilities

• Design, implement, and optimize ETL/ELT data pipelines using Apache Spark, PySpark, Databricks, or Azure Synapse.
• Build and operationalize real-time streaming pipelines leveraging Kafka, Confluent, or Azure Event Hubs for risk and liquidity data.
• Integrate and transform data across Core Banking, Trade, Payments, Treasury, CRM, and Compliance systems.
• Implement data quality, validation, and lineage controls using tools such as Great Expectations, Deequ, or dbt tests.
• Develop and maintain data models and schemas (3NF, Dimensional, Data Vault 2.0).
• Collaborate with Security and Governance teams to implement data security, masking, encryption, and tokenization in compliance with MAS TRM, PDPA, and PCI-DSS.
• Participate in data platform modernization projects (Teradata/DB2 → Snowflake, Databricks, or Synapse).
• Collaborate with Data Scientists and AI Engineers to deploy ML feature stores and model-serving pipelines.
• Support regulatory reporting (MAS 610/649) and Basel III/IV data flows.
• Maintain CI/CD pipelines for data infrastructure using Azure DevOps, Terraform, or GitHub Actions.

Required Technical Skills

• Languages: Python, PySpark, SQL, Scala
• Data Platforms: Azure Data Lake, Synapse, Databricks, Snowflake
• Orchestration: Apache Airflow, Azure Data Factory, dbt
• Streaming: Kafka, Confluent, Event Hubs
• Governance: Apache Atlas, Azure Purview, Collibra
• Security: Encryption, RBAC, Tokenization, Audit Logging
• CI/CD & IaC: Terraform, Azure DevOps, GitHub Actions

Experience and Qualifications

• 6–10 years of experience in data engineering, with at least 3 years in BFSI (banking, insurance, or capital markets).
• Proven experience building real-time and batch data pipelines on Azure or AWS.
• Exposure to regulatory data models (MAS 610, Basel III, IFRS 9/17, BCBS 239).
• Familiarity with DevOps and MLOps integration.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
• Certifications preferred: Microsoft Azure Data Engineer Associate, Databricks Data Engineer Professional, or Snowflake SnowPro Core.

Key Attributes

• Strong analytical and problem-solving mindset.
• Ability to work across multidisciplinary and geographically distributed teams.
• Excellent written and verbal communication skills.
• High accountability and ownership for quality and delivery.