Senior Data Engineer

Morgan McKinley; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Morgan McKinley in Singapore is seeking a Senior Data Engineer to architect, build, and optimize high‑throughput data pipelines powering analytics, AI, and GenAI use cases.

You will collaborate with data scientists, analysts, and system owners to translate complex requirements into secure, automated, production‑grade data platforms in a hybrid cloud environment. The role emphasizes reliability, security, release coordination, and strong ownership of production pipelines.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering or cloud-scale analytics with ownership of production pipelines.
• Strong ability to design, build, optimise, and operate production-grade batch/streaming pipelines.
• Hands-on expertise in Python and SQL for data transformation and validation.
• Familiarity with Apache Spark (PySpark) and large-scale data processing.
• Experience implementing knowledge bases and RAG solutions for AI use cases.

Responsibilities

• Build scalable ingestion, transformation, and serving capabilities across a hybrid cloud environment, ensuring reliability, security, and compliance with enterprise standards.
• Optimise batch and streaming workflows using Databricks and Kafka, applying best practices for performance, resilience, and maintainability.
• Deliver high-quality, secure data to support reporting, machine learning, and GenAI use cases.
• Lead the setup and operationalisation of knowledge bases and retrieval-augmented generation stacks for agentic AI applications.
• Perform cleansing and transformation using PySpark or SQL based on business requirements.
• Monitor, troubleshoot, and ensure pipeline reliability, while owning production readiness, observability, incident triage, and release coordination.
• Provide guidance on reusable components, code quality, deployment readiness, and production support practices.
• Connect diverse data sources (files, APIs, databases, streams), maintain metadata, and ensure transparency through documentation.
• Work with Microsoft Fabric, Databricks, Delta Lake, and other platform components.
• Drive CI/CD workflows, version control, and enforce data governance, security, and compliance policies.

Skills

Python

SQL

PySpark

Data pipelines

CI/CD

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks

Delta Lake

Kafka

Microsoft Fabric

Job description

We are seeking a Senior Data Engineer. In this position, you will architect, develop, and fine‑tune high‑throughput data systems that power scalable analytics, network telemetry processing, and advanced AI/RAG solutions. You’ll work closely with data scientists, analysts, and system owners to translate complex business requirements into secure, automated, and resilient data pipelines.

Key Responsibilities:

• Build scalable ingestion, transformation, and serving capabilities across a hybrid cloud environment, ensuring reliability, security, and compliance with enterprise standards.
• Optimise batch and streaming workflows using Databricks and Kafka, applying best practices for performance, resilience, and maintainability.
• Deliver high‑quality, secure data to support reporting, machine learning, and GenAI use cases.
• Lead the setup and operationalisation of knowledge bases and retrieval‑augmented generation stacks for agentic AI applications.
• Perform cleansing and transformation using PySpark or SQL based on business requirements.
• Monitor, troubleshoot, and ensure pipeline reliability, while owning production readiness, observability, incident triage, and release coordination.
• Provide guidance on reusable components, code quality, deployment readiness, and production support practices.
• Connect diverse data sources (files, APIs, databases, streams), maintain metadata, and ensure transparency through documentation.
• Work with Microsoft Fabric, Databricks, Delta Lake, and other platform components.
• Drive CI/CD workflows, version control, and enforce data governance, security, and compliance policies.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or related field.
• At least 5 years of experience in data engineering or cloud‑scale analytics, with proven ownership of production pipelines.
• Strong ability to design, build, optimise, and operate production‑grade batch/streaming pipelines, including orchestration, error handling, and performance tuning.
• Hands‑on expertise in Python and SQL for data transformation and validation.
• Familiarity with Apache Spark (especially PySpark) and large‑scale data processing.
• Experience implementing knowledge bases and RAG solutions for AI use cases.
• Self‑starter with problem‑solving skills, attention to detail, and ability to work independently.
• Skilled at leading technical discussions with engineers, architects, product owners, and stakeholders.
• Solid understanding of enterprise data architecture, cloud security, CI/CD, release management, and production operations for data/AI platforms.

By sending us your personal data and curriculum vitae (CV), you are deemed to consent to Morgan Mckinley Pte Ltd and its affiliates to collect, use and disclose your personal data for the purposes set out in the Privacy Policy available at https://www.morganmckinley.com/sg/privacy-policy. You acknowledge that you have read, understood, and agree with the Privacy Policy.