Senior Data Engineer

morgan mckinley pte. ltd.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Morgan McKinley is seeking an experienced Data Engineer to design, build, and maintain scalable batch and streaming pipelines on a hybrid cloud platform. You will transform data with PySpark/SQL and ensure data quality across varied sources, while guiding integration and governance practices.

You will lead RAG-enabled solutions for enterprise AI use cases, collaborate with engineers and stakeholders, and drive production readiness with strong observability and CI/CD processes.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 5-8 years of experience in data engineering, data platform engineering, or cloud-scale analytics delivery, with proven ownership of production pipelines.
• Hands-on expertise in Python, SQL, and Apache Spark (PySpark) for large-scale data processing.
• Experience building knowledge base/RAG solutions for agentic AI use cases.
• Strong understanding of enterprise data architecture, cloud security, CI/CD, release management, and production operations.
• Self-starter with strong problem-solving skills, attention to detail, and the ability to lead technical discussions across engineering, architecture, and business stakeholder groups.
• Strong documentation and communication skills.

Responsibilities

• Design, build, and operate scalable batch and streaming data pipelines across a hybrid cloud data platform.
• Perform data transformation and cleansing (PySpark/SQL) to meet business and technical requirements.
• Monitor, troubleshoot, and ensure the quality and reliability of data workflows.
• Lead integration of data from diverse sources (files, APIs, databases, streaming platforms).
• Lead the implementation of knowledge base and Retrieval-Augmented Generation (RAG) solutions to support GenAI use cases.
• Provide technical guidance to engineers and delivery partners on platform patterns and production support practices.
• Own production readiness for assigned components—observability, incident triage, root-cause analysis, and runbook improvements.
• Maintain metadata/pipeline documentation and contribute to CI/CD automation efforts.
• Apply data governance, security, and access control policies throughout design and implementation.

Skills

Python

SQL

Apache Spark (PySpark)

Databricks

Kafka

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks

Kafka

PySpark

Job description

Company Summary

A leading telecommunications company has launched a new business unit dedicated to Artificial Intelligence & Data Analytics - a strategic engine driving enterprise-wide AI transformation. This unit is tasked with embedding intelligence into every layer of the business, aligning people, platforms, and processes to build AI literacy and create a culture where intelligence augments human potential. Joining this team means contributing to a company-wide shift reshaping the telecom industry through AI-driven innovation.

Roles & Responsibilities

• Design, build, and operate scalable batch and streaming data pipelines (using tools like Databricks and Kafka) across a hybrid cloud data platform, ensuring reliability, security, and alignment with enterprise architecture standards.
• Perform data transformation and cleansing (PySpark/SQL) to meet business and technical requirements.
• Monitor, troubleshoot, and ensure the quality and reliability of data workflows.
• Lead integration of data from diverse sources (files, APIs, databases, streaming platforms), coordinating with source-system owners and consuming teams.
• Lead the implementation of knowledge base and Retrieval-Augmented Generation (RAG) solutions to support scalable GenAI and agentic AI use cases, including storage, lifecycle management, and embedding/vectorization.
• Provide technical guidance to engineers and delivery partners on platform patterns, code quality, and production support practices.
• Own production readiness for assigned components - observability, incident triage, root-cause analysis, and operational runbook improvements.
• Maintain metadata/pipeline documentation and contribute to CI/CD automation efforts.
• Apply data governance, security, and access control policies throughout solution design and implementation.

Requirements

• Bachelor's degree in Computer Science, Engineering, or related field.
• 5-8 years of experience in data engineering, data platform engineering, or cloud-scale analytics delivery, with proven ownership of production pipelines.
• Hands-on expertise in Python, SQL, and Apache Spark (PySpark) for large-scale data processing.
• Experience building knowledge base/RAG solutions for agentic AI use cases.
• Strong understanding of enterprise data architecture, cloud security, CI/CD, release management, and production operations.
• Self-starter with strong problem-solving skills, attention to detail, and the ability to lead technical discussions across engineering, architecture, and business stakeholder groups.
• Strong documentation and communication skills.